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Background Information About 
the SERVE Center 


The SERVE Center at the University of North Carolina 
at Greensboro (UNCG) is a university-based research, 
development, dissemination, evaluation, and technical 


assistance center. Its mission is to support and promote 


teaching and learning excellence in the education community. 


Since its inception in 1990, SERVE has been awarded over 
$200 million in contracts and grants. It has successfully 
managed 16 major awards, including four consecutive 
contracts for the Regional Educational Laboratory for the 
Southeast (REL-SE) funded by the Institute of Education 
Sciences (IES) at the U.S. Department of Education (USED), 
and five awards from USED for the National Center for 
Homeless Education (NCHE). In addition, past SERVE awards 
include a five-year Technology Grant for Coordinating Teaching 
and Learning in Migrant Communities, three consecutive 
contracts as the Eisenhower Consortium for Mathematics 

and Science Education for the Southeast, and two consecutive 
Regional Technology in Education Consortium grants. 


SERVE is currently hosting a U.S. Department of Education 
Regional Center, providing services to North Carolina, South 
Carolina, and Georgia. At the national level, SERVE operates 
the National Center for Homeless Education (NCHE), USED’s 
technical assistance and information dissemination center in 


the area of homeless education. 


In addition to national-level NCHE activities, SERVE 
currently conducts research studies and evaluations under 
grants and contracts with federal, state, and local education 
agencies. Examples of SERVE’s grant-funded research work 
include three federally funded studies of the impact of Early 
College High Schools, and a five-year impact study of North 
Carolina’s dual enrollment program. Contract work includes 
evaluations of five Investing in Innovation (13) projects, the 
Winston-Salem/Forsyth County Magnet Program in North 
Carolina, the Guilford County Schools teacher incentive 
program (Mission Possible), the USED-funded Bridges to 
Early Learning Project in South Carolina, and North Carolina’s 
Race to the Top Initiative. The Program Evaluation Standards, 
Second Edition (‘The Joint Committee on Standards for 
Educational Evaluation, 1994), the Guiding Principles for 
Evaluators (American Evaluation Association, 2004), and the 
What Works Clearinghouse Standards (Institute of Education 
Sciences, 2018) guide the evaluation work performed at the 
SERVE Center. 
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Section I: Background and Introduction 


Columbus State Community College has a long history of 
working to increase the skills and expertise of the workforce 
in the Central Ohio area. Through an initiative known as the 
Central Ohio Compact, Columbus State and a broad range of 
partners have engaged in a comprehensive set of strategies 
with the ultimate goal of building a workforce with the 
education and career-preparatory experiences necessary to 
support business and industry in the Columbus region. 


Within this broader initiative, the College and Career 
Readiness Expansion (CCRE) project was intended to focus 
on the K-12 sector. Supported by a $11.6 million grant from 
the U.S. Department of Education’s Investing in Innovation 
(i8) program, the Career and College Readiness Expansion 
project sought to support economic development in the 
Columbus, Ohio area by increasing the number of students 
who graduated from high school and who were prepared for 
enrollment and success in postsecondary education. The 
project did this through implementation of the early college 
high school model in comprehensive high schools across 
seven school districts. 


The early college high school (early college) model is a 
research-based strategy to combine aspects of the high 
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school and college experiences. Early colleges have often 
been implemented as small schools on college campuses with 
the initiatives primarily led by school districts or non-profit 
organizations. CCRE experimented with this approach by 
implementing the model in large comprehensive high schools 
and by having a community college lead the work. More detail 
on the model and the supports provided by Columbus State 
and its partners is provided in the next section. 


This report presents final results from the impact study 

of CCRE. It includes the results of a quasi-experimental 
study that examined the impact of the project on a core set 
of student outcomes. It also includes descriptive findings 
about school-level changes that accompanied CCRE 
implementation. A summary of the findings can be found 
in the executive summary that accompanies this report. 
This longer report includes more detail and begins with 

an overview of the CCRE model in Section II. Section II 
presents the technical details of the study methodology. 
Section IV presents findings regarding changes in 
participating schools and districts, which is followed by 
the impacts on students in Section V. The final section of 
the report discusses implications of this work and presents 
lessons learned for future replication. 
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Section II: The CCRE Program 


The College and Career Readiness Expansion Project 

was implemented by Columbus State in conjunction 

with two key partners: 1) Jobs for the Future (JFF), a 
national organization focused on policies and interventions 
supporting the transition to college and career; and 2) the 
Educational Service Center of Central Ohio (ESC), a regional 
educational agency that supports initiatives improving school 
effectiveness and student achievement. Under the CCRE 
model, Columbus State and its partners worked with seven 
districts in the Central Ohio area to implement strategies 
from the early college model within comprehensive high 
schools. This section describes the CCRE model, the activities 
that supported the implementation model, and the internal 
changes Columbus State made to do this project. 


I1.1. HISTORY OF THE EARLY COLLEGE MODEL 

As originally conceptualized, early colleges are small schools 
that integrate the high school and college experiences. 
Frequently located on college campuses, early colleges 
target students who are underrepresented in postsecondary 
education. Early colleges serve students starting in ninth 
grade; the goal is to have students graduate in four or five 
years with a high school diploma and either a postsecondary 
credential (usually an associate degree) or two years 

of transferable college credit. Supported by an initial 
investment by the Bill and Melinda Gates Foundation, the 
small early college model expanded across the country. 


This model has been the subject of two rigorous experimental 
studies, a 14-year longitudinal experimental study led by the 
SERVE Center at UNCG and a retrospective experimental 
study conducted by the American Institutes of Research. 
These studies found that the early college model had positive 
impacts on a variety of outcomes, including staying in school, 
progressing in college preparatory courses, graduating from 
high school, and enrolling in and graduating from college 
(Berger et al., 2013; Edmunds, Bernstein, Unlu, Glennie, 


thttps://www.ohiohighered.org/ccp/faqs 


& Smith, 2013; Edmunds et al., 2012; Edmunds et al., 2017; 
Edmunds, Willse, Arshavsky, & Dallas, 2018; Haxton et al., 
2016; Song & Zeiser, 2019). 


11.2. THE CCRE PROJECT SCHOOLS 

AND STATE CONTEXT 

CCRE was implemented in 16 high schools across seven 
districts. Columbus State aimed to include districts where 

at least 40% of students were considered economically 
disadvantaged. In six out of the seven districts, all of the high 
schools in the district were served by the grant. Four districts 
were small districts with only one high school. Two of the 
districts had four high schools with all participating in the 
project. One district was a large urban district and had four of 
its high schools participate in the project. 


The participating schools and districts represented the 
geographic diversity of the Central Ohio area. Using NCES 
locale codes from the Common Core of Data, four (25%) were 
urban, nine (56%) were suburban, and three (19%) were rural. 
The CCRE schools differed greatly in size, ranging from a few 
hundred students to close to 2,000 students with an average 
of approximately 950 students. 


CCRE was implemented within the context of Ohio’s College 
Credit Plus (CCP) policy. Under the policy, which came into 
effect during the 2015-16 school year, Ohio students in grades 
7-12 can apply to any public college or university in Ohio as 
well as one of several participating private institutions in 
the state.' After meeting the entrance requirements for the 
institution, such as a qualifying score on a placement test, 
students may enroll in college courses at public institutions 
and earn transferable college credit. Transportation for 
courses that meet face-to-face is not provided, however, 
creating the necessity for more courses to be offered on 

high school campuses. 
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11.3. THE CCRE MODEL 

The CCRE model was based on the early college model and 
included many of the same elements (or Design Principles); 
a key difference was that CCRE was implemented in 
comprehensive high schools as opposed to in small schools 
of choice. Small early colleges expect all of their students 
to earn an associate degree or two years of college credit 

at the same time as their high school diploma. Because 
comprehensive schools serve a wider range of students, 

the CCRE model had to modify this expectation; however, 
it had a still ambitious goal of having 90% of high school 
students graduate with at least three hours of college credit 


or a career credential. 


11.3.1 Design Principles 

To support this goal, the expectation was that participating 
high schools would implement four Early College Design 
Principles: 1) a College-Ready Academic Program, 2) a College 
Headstart, 3) Wraparound Student Supports, and 4) School- 
level Organizational Practices. Each is briefly described 
below and in the section that discusses the impact on schools 
(Section V). 


Career- and College-Ready Academic Program 

A College-Ready Academic Program included an academic 
program of study that allowed almost all students to be 
prepared for college and to attain college credit while still 
in high school. To do this, schools were expected to expand 
opportunities for students to earn at least three college 
credits. Part of this expansion included the creation of 
pathways that focused student coursetaking, so that the 
courses could contribute to a major or a credential. These 
pathways were intended to include work-based learning 
experiences when appropriate. Finally, this Design Principle 
also focused on classroom practices and instructional 
strategies that enhance rigor, including the Common 
Instructional Framework (CIF). 


College Head Start 
The second CCRE Design Principle, College Headstart, 
focused on providing students with early exposure to the 
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culture and norms of college. This could be done through 
college readiness skills instruction or through college 
readiness support activities, such as advising on the 
courses needed for college and taking students to visit 
college campuses. 


Wraparound Student Supports 

The Wraparound Student Supports Design Principle 
included comprehensive academic supports, social and 
emotional programming and support, and assistance 

with college applications and seeking financial aid. These 
supports involved school staff developing and sustaining 
relationships with students, providing academic assistance 
outside of regular class time, and employing systems that 
identify student needs and suggest targeted interventions. 
Logistic supports, such as registering for placement tests and 
courses, navigating college procedures, and understanding 
how to make use of college resources, also fell within the 
Wraparound Student Supports Design Principle. 


School-Level Organizational Practices 

The fourth CCRE Design Principle encompassed school- 
level practices that needed to be in place to ensure effective 
implementation of the other Design Principles. These 
practices entailed: 1) development of structures to support 
personalized relationships; 2) establishment of a college- 
going culture; 3) ongoing job-embedded professional 
development; 4) data-based decision-making; and 5) time and 
support for teacher collaboration. 


11.4. IMPLEMENTATION SUPPORTS 

The CCRE model included a suite of services (known as Key 
Components) intended to support a school’s implementation 
of the Design Principles described above. The following 

Key Components were the responsibility of Columbus State 
Community College and its partners, Jobs for the Future 
and the Educational Service Center of Central Ohio: 1) 
creation of a regional governance structure, 2) professional 
development and coaching for district and school staff, 3) 
curriculum development and alignment, 4) professional 
development for college faculty, and 5) student support 
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Figure Il-1: Career and College Readiness Expansion Logic Model 
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activities. The Key Components also included the following 
activities completed by the local school districts: 6) creation 
of a strategic implementation plan, 7) leadership coaching for 
principals and selected administrative staff, 8) professional 
development for school staff, and 9) creation of integrated 
9-14 pathways. Each support is described in more detail 
below. Figure II-1 presents the relationship between the Key 


Components, the Design Principles, and the intended student 


outcomes. 


The evaluation team worked with the project staff to identify 
threshold levels of implementation for each of the program 
activities (measure of Fidelity of Implementation, or FOI). A 
summary of FOI for Years 2 and 3 is presented in Appendix A. 
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Because of challenges due to the COVID-19 pandemic, FOI is 
not reported for Year 4. 


It is important to note that although there were common 
supports offered to all districts, the nature and content of 
these supports were often customized to the districts’ needs. 
As one Columbus State staff member noted, “One of the 
most glaring lessons that I think not just we learned as a 
project team, but also myself, is that because we have seven 
very unique partners there needs to be seven very unique 


approaches to how you work with them.” 


11.4.1 SUPPORTS PROVIDED BY 

EXTERNAL PARTNERS 

Some of the implementation supports (the Key Components 
in the first column in Figure 1) were provided by Columbus 
State and its partners, JFF and the ESC. Highlights of these 
supports are included below; more detailed resources can be 
found in the CCRE Toolkit. 


Governance Structure 

The project’s governance structure evolved over the course 
of the grant. From the beginning, the project had an “i3 
Cabinet,’ which included senior representatives from 
participating districts, usually the superintendent, and which 
was designed to guide and make decisions about the overall 
work of the grant. These meetings were supplemented by 
Core Team meetings, which included college and district 
staff, and which focused on the details of implementing 
project activities. By the final year of the project, there were 
fewer centralized meetings and more “one-on-one” meetings 
between Columbus State and one district’s representatives to 
address the districts’ specific needs. 


Each district was also allowed to hire a project coordinator, 
an “13 coordinator,’ to guide the project’s implementation. 
One district representative commented that they were only 
able to make as much progress as they did because they had 
a staff member dedicated to moving the career and college 
readiness work forward. 
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Technical Assistance, Professional Development, 
and Coaching 

To support project implementation, the project partners 
provided a range of capacity development activities to district 
and school staff. At the beginning of the project, there was 

an expectation that districts and schools participate in 
leadership and instructional coaching provided by JFF. As 
the project evolved, the coaching activities evolved to be 
more customized to each district and the ESC took on more 
coaching responsibilities. Leadership coaching involved 
working with district and school leaders to assist them in 
implementing the CCRE model; project data showed that 
100% of principals reported working with a leadership coach. 
Instructional coaching emphasized the incorporation of 
instructional practices expected to prepare students for 
success in postsecondary education; survey data showed that 
approximately half of the teachers reported working with an 


instructional coach. 


In addition to the coaching, the districts and schools could 
participate in a variety of professional development activities, 
such as workshops or trainings, on a range of topics related to 
the foci of the grant. Many of these sessions were provided by 
the ESC. In some cases, professional development included 
attending conferences (e.g., Success Network Meeting, 

Big Think Conference 2019); in other cases, professional 
development happened in district offices or schools. Topics 
included student success plans, utilizing Naviance (a college 
and career advising system), College Credit Plus updates, 
developing apprenticeship plans, and implementing a 
multi-tiered system of support (MTSS). By the end of the 
grant, there was an increased emphasis on topics related to 
sustaining and spreading the work. 


Curriculum Development and Alignment 
Columbus State provided districts with technical assistance 
around developing pathways that aligned high school and 
dual-enrollment courses such that these courses led to a 
credential or a degree. Over the course of the project, the 
technical assistance evolved to not only help plan pathway 
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coursework but also to address issues like finding qualified 
staff to teach courses, planning for student exploration in 
pathways, holding collaboration meetings for staff members 
from different institutions, aligning pathways with other 
programming in schools, and solving other logistical issues. 


Additionally, Columbus State adapted an existing one-credit 
college success course (COLS 1101) for implementation in 
the high school environment, making the content more 


“high school-centric” and including three projects designed 
to help students with future planning. Districts further 
adapted this course to their own needs, with some districts 
integrating its content into regular high school courses and 
others offering it as a stand-alone course that would give 
students college credit. 


Columbus State also partnered with districts to create a class 
called “Third Space English” that built students’ college- 
course readiness. A district coordinator described the effort: 


“It's called Third Space because it's not 
the high school space, it's not the college 
space, but it's a third space, where the 
high school and college professors and 
teachers have come together, developed 
out a semesterized course that really acts 
as a remedial English course.” 


—District Coordinator 


At the end of the project, Columbus State had received a 
waiver from the Ohio Department of Higher Education that 
allowed them to treat this course as an alternative way of 
demonstrating readiness for the first level of college English. 


Professional Development for College Faculty 
Columbus State provided professional development for two 
groups of college faculty: 1) college instructors who taught 
dual-enrollment courses, and 2) high school instructors 


who served as adjunct faculty for the college and taught 
dual-enrollment courses. Dual enrollment instructors were 
expected to attend an orientation (when they were new) 
and additional professional development sessions that were 
often sponsored by the departments with which they were 
associated. As shown in Appendix A, very few instructors 
participated in the expected two sessions of professional 
development annually. The Columbus State staff noted that 
the development of the Third Space English (described 
above) was one of the most effective ways of developing 
faculty skills. 


Student Support Activities 

Columbus State provided a variety of student support 
activities to students taking dual-enrollment courses. First, 
they had dedicated staff that provided academic advising to 
students registering for and taking dual-enrollment courses. 
They started with four on-site advisors (who worked in the 
high schools), but in an effort to make the advising more 
sustainable, shifted to fewer advisors with more online 
support. Second, the college provided each district with data 
from their Starfish system that monitored how students were 
doing in their college classes. Each high school had a Starfish 
contact who was trained in how to use the data. The final 
support was tutoring for students who were taking college 
classes. Dual enrollment students could take advantage of 
all on-site supports at the college, as well as online tutoring 
services, which was used more frequently by students. All of 
these Columbus-state supported activities were implemented 
as intended. 


11.4.2 DISTRICT-PROVIDED SUPPORTS 

Districts were also expected to provide supports to schools 
implementing the early college model. These supports are 
discussed in this section. 


Strategic Plans 

Each district was expected to create a strategic 
implementation plan to guide implementation of the 
Early College Design Principles. At the beginning of the 
project, Columbus State and the ESC collaborated with 
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the districts to create a template of program activities and 
prioritize areas of focus for each implementation year. As the 
project evolved, the grant moved from expecting a separate 
plan to having them integrate program activities into their 
regular district plans. 


Leadership Development 

As a strategy to ensure sustainability, each district was 

also supposed to provide professional development to 
school leadership. Each district had a coordinator who was 
responsible for managing the project within the district. In 
addition, each district was expected to provide an average 
of 12 hours of professional development a year for their 
principals. In Year 2, 10 of the 16 schools met that threshold 
and in Year 3, half of the schools received the expected 
number of hours. 


Professional Development for School Staff 
Districts were also expected to provide project-relevant 
professional development to teachers, counselors, and other 
school staff. These topics varied by district. For example, in 
one district, the coordinator provided support to teachers on 
different i3 topics each week. Another district focused on CTE 
planning and support, while a third emphasized multi-tiered 
systems of support, curriculum mapping, and data review. 
The expectation was each school receive at least 12 hours of 
professional development annually. In Year 2, this threshold 
was met by 12 out of 16 schools and in Year 3 by 11 schools. 


Creation of Pathways 

Acore activity of the project was creating integrated 
pathways, which were intended to blend high school and 
college coursework to lead to a credential or degree program. 
By the end of the project, each district had designed at 

least one pathway, although the level of integration of high 
school and postsecondary coursework varied across districts. 
There was also an expectation that work-based learning 
should be integrated into the pathways as appropriate; at 
the end of the project fewer than half the districts had an 
explicit, intentional connection between work-based learning 


activities and stated pathways. 
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11.4.3 CHANGES MADE AT COLUMBUS STATE 

As described above, Columbus State provide an extensive 
amount of support to the schools. The College took the 
opportunity to learn from the CCRE Project and reconsider 
how they interacted with the K-12 system and to reorganize 
themselves. For example, at the start of the project, there 
was a dual enrollment office that operated separately from 
other units at the College. As the number of dual enrollment 
students increased, the College realized that this work 
needed to be embedded throughout the normal college 
processes. Thus, sections like Enrollment Management 
and Student Services took on a more active role in dual 
enrollment administration. As one staff member said, 
“Whether it's admissions, whether it's advising, whatever 
that might be, it's become an integral part of the College’s 
work.” The college also hired a dual enrollment coordinator 
who reports directly to the Chief Academic Officer. 


Columbus State was also able to use the CCRE project to 
experiment with different models of student support and 
advising. As a result of the project, Columbus State has also 
institutionalized practices such as providing data from early 
alert systems to high schools. More information on how the 
college has evolved can be found in a policy brief written by 
Columbus State. 


The CCRE evaluation consisted of three different sub-studies: 
1) a quasi-experimental impact study using administrative 
data from the Ohio Department of Education, 2) an annual 
survey concerning program implementation that was 
administered to staff in treatment schools and a set of 
comparison principals, and 3) case studies of six program 
schools. This section describes the methodology used for each 
of these three sub-studies. 
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Section III: Methodology 


11.1. ESTIMATING THE IMPACT ON STUDENTS 
The impact study portion of the evaluation investigated the 
following research questions: 


1. DoCCRE high schools have a higher percentage 
of students on-track in college preparatory course 
completion in ninth grade as compared to comparison 
schools? 


2. DoCCRE high schools have a higher percentage 
of students on-track in college preparatory course 
completion in tenth grade as compared to comparison 
schools? 


38. Do CCRE high schools have fewer students dropping out 
than the comparison schools? 


4. DoCCRE high schools have a higher percentage 
of students enrolling in college courses than the 


comparison schools? 


5. Dostudents in CCRE high schools receive more college 
credits than students in the comparison schools? 


To determine the impact of the program, the evaluation used 
a quasi-experimental design in which participating schools 
were matched to similar, non-participating schools, and the 
outcomes of interest were compared. 


I11.1.1. School Sample 

Under the quasi-experimental framework, we identified a 
group of comparison schools that closely matched the CCRE 
program schools (treatment schools) on various demographic 
and performance variables the year before the program 
began. Using a comparison group allowed us to determine the 
impact of the program based on the differences in outcomes 
between schools who participated in CCRE versus other Ohio 
high schools not participating in the program. 


‘We determined comparison group schools using the genetic 
matching algorithm from the MatchIt package in R. Each 
of the 16 treatment schools was matched to two comparison 


schools (for a total of 32 comparison schools). We selected 
the comparison schools from a pool of 394 schools; this pool 
excluded schools within Columbus State’s service region, 
charter schools, small stand-alone early college high schools, 
alternative schools, and schools with missing data. We 
derived all school-level variables from the Ohio Department 
of Education’s (ODE) administrative data from 2015-16, 

the last school year prior to the beginning of the CCRE 
program. All student-level variables were determined using 
students’ eighth grade data. Table III-1 (page 13) summarizes 
all variables used in the matching procedure under the 
domains of demographics, pre-high school student academic 
performance, and school-level baseline outcomes. The final 
analytic sample contained 48 schools (32 comparison and 

16 treatment). 


We also used student-level measures to assess baseline 
equivalence at the student level and to include in the 
analyses. Table III-2 (page 15) lists these variables; the first 
four were used for student-level baseline equivalence and all 


were incorporated into the impact models. 


Baseline Equivalence 

‘We conducted analyses to demonstrate the extent to which 
treatment and comparison schools were similar at baseline 
on school-level measures of school baseline characteristics. 
Those results are presented first. We also conducted analyses 
to demonstrate the extent to which the treatment and 
comparison students used in each of the impact analyses 
were similar to one another on student-level baseline 
characteristics. Those results a presented after the results on 
the baseline equivalence of schools. 


School-level baseline equivalence. We began by assessing 
baseline equivalence at the school-level using data from two 
primary sources: 1) publicly available measures of school 
demographics and 2) baseline outcome measures calculated 
from restricted-use student-level data. 


serve.uncg.edu 


THE IMPACT OF THE COLLEGE AND CAREER READINESS EXPANSION (CCRE) PROJECT | 13 


Table III-1. Definitions of Variables used for Matching and Baseline Equivalence 


School Demographics (Matching and baseline equivalence conducted at the school level) 


“Underrepresented race/ethnicity” includes students who 

are members of racial or ethnic groups traditionally 
underrepresented in college. These include the percentage of Black, 
Hispanic/Latino, Multiracial, and American Indian/Alaska Native 
students, calculated from the public Ohio School Report Card 
Building Race/Ethnicity file. 


Percent Underrepresented Race/Ethnicity (2015-16) 


Low-income students are a target population of the intervention. 
This variable includes the percentage of students who are 

Percent Economically Disadvantaged (2015-16) designated in the state data as “economically disadvantaged,” 
calculated from the public Ohio School Report Card Building 
Economically Disadvantaged file. 


The proportion of students with disabilities provides evidence of 
special populations within a school who may receive additional 
services through their high school. This figure was calculated from 
data in the ODE student-level Enrol/ment file as the proportion of 
students with one or more disability codes. 


Percent of Students with One or More 
Disabilities (2015-16) 


. This is a measure of school size, calculated from the public Ohio 
Number of Students Enrolled in Grades 9-12 (2015-16) ten i 
School Report Card Building Enrollment file. 
In the 16 CCRE treatment group schools, 15 serve grades 9-12 and one 
serves grades 7-12. This variable indicates whether a school serves 
middle grades (7-8), with data coming from the public Ohio School 
Report Card Building Enrollment file. 


School Serves Grades 7-12 (2015-16) 


School-Level Baseline Academic Performance (Matching conducted at the school level) 


This school-level baseline measure was created using 

student-level 8th grade math scores from the ODE student-level 

Assessment file. The largest share of students took the Math 8 EOG, 
Mean 8th Grade Math Z-Score in 2015-16 for 2016-17 the Algebra | EOC, or the Math | EOC in 2015-16, but some students 
9th Grade Cohort (Used only for matching) in the sample took Geometry or Math II in 2015-16. A z-score was 

calculated for each student’s performance relative to all other test 

takers in the state for a given subject and year. A mean score was 

calculated for the school. 


This school-level baseline measure was created using students’ 8th 
grade ELA scores from the ODE student-level Assessment file. The 


Mean 8th Grade ELA Z-Score in 2015-16 largest share of students took the Reading 8 EOG in 2015-16, but 
for 2016-17 9th Grade Cohort (Used only for some students in the sample took English | in 2015-16. A z-score was 
matching) calculated for each student's performance relative to all other test 


takers in the state for a given subject and year. A mean score was 
calculated for the school. 
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Table III-1. Definitions of Variables used for Matching and Baseline Equivalence (continued) 


School-Level Baseline Outcomes (Matching and baseline equivalence conducted at the school level) 


This is a baseline measure at the school level for the 2015-16 cohort 
of 9th grade students, one of the outcomes examined in the study. 
The measure was calculated using the same criteria as that for the 
9th grade on-track outcome,” using graduation points earned by 
students in the ODE student-level Assessment file. 


Percent of On-Track Students (Using EOC Point 
Definition) for 2015-16 9th Grade Cohort 


The extent to which students persist in school is one of the outcomes 
examined in the study. The school-level dropout 

rates from the 2015-16 school year were calculated using the ODE 
student-level Dropout file. The baseline outcome was calculated using 
the same procedure as the one used in calculating the outcome for 
the analytic sample. 


Dropout Rate in Grades 9-12 in 2015-16 


The percentage of students taking college courses in high school 

is an outcome of the study. This is a baseline measure at the 

school level for the 2015-16 cohort of 10-12 grade students for 
dual-enrollment courses through the College Credit Plus (or CCP) 
program. The measure was calculated using the ODE student-level 
Courses file. If a student had at least one course record for CCP (in 
Ohio administrative data as ‘PS’), the variable was coded as aland0O 
otherwise. The school-level measure is the proportion of students in 
the sample coded as a1 for CCP coursetaking. 


Percent of Students in Grades 10-12 Taking? One or 
More Dual-enrollment courses (CCP) in 2015-16 


The percentage of students taking college courses in high school 

is an outcome of the study. This is a baseline measure at the school 

level for the 2015-16 cohort of grade 10-12 students for Advanced 
Percent of Students in Grades 10-12 Placement (AP) courses taken. The measure was calculated using 
Taking One or More AP Courses in 2015-16 the ODE student-level Courses file. If a student had at least one 

course record for an AP class, the variable was coded as al and O 

otherwise. The school-level measure is the proportion of students in 

the sample coded as a1 for AP coursetaking. 


The mean number of college credits is an outcome of the study. 
This is a baseline of the school-level mean number of college credits 
earned by students in the Class of 2016. At the student level, the 
number of dual credits was determined by the value in the ODE 
student-level Grad Core file. The number of equivalent college 
credits earned through AP was calculated using ODE student-level 
AP test scores cross-walked with the number of college credits 
corresponding to the AP score for Ohio University. 


Mean College Credits Earned by Class of 2016 


?More details on the on-track definition are provided in section IIL1.2. 


’More details on why Grades 10-12 were used for the college coursetaking sample are offered in section IL.1.2.4 of this report. 
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Table III-2. Definitions of Student-Level Variables Used in Analyses 


(o}Vi Xovoyant-W- Tale M ozo) oLUl tafe) a} CCRE Adjusted Mean 


This is a baseline measure at the student level for students’ 8th grade math 
score from the ODE student-level Assessment file. A z-score was calculated 
for each student's performance relative to all other test takers in the state for a 
given subject and year. 


Baseline Math Z-Score 


This is a baseline measure at the student level for students’ 8th grade ELA score 
from the ODE student-level Assessment file. A z-score was calculated for each 
student's performance relative to all other test takers in the state for a given 
subject and year. 


Baseline ELA Z-Score 


This is a student-level measure from the Enrollment file for economically 

disadvantaged status in the Enrollment file in a) a student’s 8th grade year or b) 
Economically Disadvantaged Status the most recent year available if the student's 8th grade record is missing. 

The variable is coded as 1 if the source data value ECON_DIS is “Y” and 0 if 

ECON_DIS is “N” in the year of interest. 


This is a student-level measure from the Enrollment file for race/ethnicity 
membership in a group underrepresented in higher education in a) a student's 
8th grade year or b) the most recent year available if the student’s 8th grade 
record is missing. This variable is coded as 1 if the source data value of 
RACE_CODE is in the set {“B”, “H”, “I”, “M”, “P”} and 0 if the value of RACE_CODE 
is in the set {“A”, “W”}. 


Underrepresented Race/Ethnicity Status 


This is a student-level measure from the Enrollment file for limited English 
Limited English Proficient Status proficient status in the current school year. The variable is coded as 1 if the 
source data value of LEP_CODE is “N”and 0 for all other non-missing values. 


This is a student-level measure from the Enrollment file for disability status in the 
Disability Status current school year. The variable is coded as 1 if the source data value of DISAB_ 
CODE is “**” (indicating no disability codes) and O for all other non-missing values. 


This is a student-level measure from the Enrollment file for gender in a) a 
student's 8th grade year or b) the most recent year available if the student’s 


Pender 8th grade record is missing. The variable is coded as 1 if the source data value of 
GENDER_CODE is “M” and 0 for a GENDER_CODE of “F.” 

For the school-level demographic measures shown in Table to the weighted treatment and comparison means. For 
III-3 (page 16), we used school-level data from the Ohio dichotomous variables, we calculated the effect size using 
Department of Education. Because we wanted to account for Cox’s Index. For the continuous variable of enrollment, we 
the fact that the schools were of varying sizes,* we weighted calculated the effect size using Hedges’ g as the weighted 
the treatment and comparison means by the total grade 9-12 difference in means divided by the weighted standard 
enrollment in a school. In other words, we multiplied the deviation. The formula for weighted standard deviation is as 
school-level percentages by the number of students enrolled follows — for each school i, the enrollment n and the overall 
in Grades 9-12 in 2015-16 to determine their contribution mean U: 


*The weighting also allowed us to replicate the results that would have occurred if we had created these measures using student-level data as was done with the baseline measures of the outcomes and as was done with the impact analyses. 
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Table III-3. School-Level Baseline Equivalence — Baseline School-Level Demographics 


Weighted 
Comparison 
Mean 


Weighted 


Treatment 


Variable Treat N Mean 


School-Level Demographics (calculated from school-level baseline data) 


Pct Underrepresented Race/ 15,416 30,999 
woe 44.5% 46.7% -.08 
Ethnicity (2015-16) (16 sch) (32 sch) 
Pct Economically 15,416 30,999 
: 60.9% 59.7% 04 
Disadvantaged (2015-16) (16 sch) (32 sch) 
Pct Students with a 15,416 30,999 
sh ocawiys 14.5% 14.9% -.08 
Disability (2015-16) (16 sch) (32 sch) 
Percentage of Schools with 
16 sch 32 sch 3.7% 6.6% -.12 
Grades 7-12 (2015-16) 
Enrollment in 15,416 30,999 969 964 o1 
Grades 9-12 (2015-16) (16 sch) (32 sch) (SD=468) (SD=404) : 
(In, (&)-)” 


Weighted SD = 
2n-1 


For the baseline outcomes calculated from student-level 
baseline data shown in Table III-4 (page 17), we first used a 
hierarchical linear model (HLM) with the covariate as the 
outcome, a treatment indicator, and a school-level random 
intercept to calculate the difference between the treatment 
and comparison students. For the baseline outcomes 
calculated from student-level baseline data shown in Table 
IIT-4, we calculated the group means for students attending 
the treatment and comparison schools. For dichotomous 
outcomes, we calculated Cox’s Index for the treatment mean 
and comparison mean. For the continuous outcome, we 
divided the difference of the treatment and comparison means 
by the pooled standard deviation to calculate Hedges’ g. 


As the tables show, the two groups of schools were equivalent 
on all measures with effect sizes of .14 SD or less. 


Because we used baseline measures from a previous cohort 
of students, we also assessed the representativeness of each 
sample for each outcome. The sample criteria are described 
separately for each outcome. We used the same approach for 
creating the samples for the baseline measures as well as for 
the outcome measures. Our analyses of representativeness 
for all samples showed that the study met What Works 
Clearinghouse (WWC) Standards for representativeness. 
Tables for each sample can be found in Appendix B. 


Student-Level Baseline Equivalence. The school-level baseline 


equivalence above demonstrates that the schools were 
equivalent at baseline. As an extra check on our sample, we 
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Table III-4. School-Level Baseline Equivalence — Baseline Outcomes from Student-Level Data 


Treatment Comparison 


Variable Treat N Comp N Mean Mean 


Baseline Outcomes (calculated from student-level baseline data) 


Pct of On-Track Students 
. . ae 3,918 7,454 
(Using EOC Point Definition) 471% 49.2% -.05 
(16 sch) (32 sch) 
for 2015-16 9th Grade Cohort 


Dropout Rate in Grades 14,373 28,389 

: ; , 3.3% 3.1% 04 
9-12 in 2015-16 (16 sch) (32 sch) : : 
Pct of Students in Grades 
10-12 Taking One or More 10,603 22,161 9.3% 9.0% 02 
Dual-enrollment courses (16 sch) (32 sch) ye ae ; 


(CCP) in 2015-16 


Pct of Students in Grades 10,603 22,161 

as H 9 0, 
10-12 Taking One or More AP (16 sch) (32 sch) 14.5% 11.8% 14 
Courses in 2015-16 


Mean College Credits Earned by 2,665 6,039 2.45 2.51 Ol 

Class of 2016 (16 sch) (32 sch) (SD=5.85) (SD=6.62) 
also assessed baseline equivalence at the student level for students and economically disadvantaged students. It is 
the analytic sample of students defined for each outcome. We important to note that there were some schools that did 
checked student-level baseline equivalence for two primary not have any non-economically disadvantaged students or 
demographic characteristics, economically disadvantaged non-underrepresented race/ethnicities; as a result, those 
status and race/ethnicity, as well as middle school sub-groups in the tables below include a different number of 
achievement measures in ELA and math. We calculated schools. Student-level baseline equivalence for the analytic 
the difference between the treatment and comparison sample for each outcome measure is presented below for each 
schools using the coefficient of a simple hierarchical linear outcome measure. 
model (HLM) model to account for the nesting of students 
within schools with the baseline equivalence variable (such I11.1.2, Outcome Measures, Analytic Sample, and 
as economically disadvantaged) as the outcome, a binary Student-level Baseline Equivalence 
treatment indicator, and a random intercept for the school. The study assessed the impact of CCRE on a core set of high 
The coefficient on the treatment indicator represents the school student outcomes: 1) students’ successful progression 
difference. We then calculated Cox’s Index for the effect ina college preparatory course of study, 2) the number of 
size on the dichotomous outcomes and Hedges’ g (the students staying in school, and 3) student enrollment and 
coefficient divided by the pooled standard deviation) for the success in college-level courses. The measure, student-level 
continuous outcomes. We replicated these analyses for two sample for that measure, and modifications to the general 
different sub-groups: racially/ethnically underrepresented analytic approach are discussed below. 
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111.1.2.1. Ninth Grade On-Track in 2017-18 and 2018-19 
Outcome 1 measured the extent to which students were on- 
track for high school graduation in sample schools by the end 
of their ninth grade year, determined by graduation points 
students earned on End of Course (EOC) tests. This outcome 
was measured for two cohorts, ninth-graders in 2017-18 

(Year 2 of full implementation) and in 2018-19 (Year 3 of full 
implementation). 


Outcome Variable. We designed Outcome 1 to assess the 
extent to which ninth grade students made academic progress 
that would allow them to graduate from high school prepared 
for career or college. In other studies, we have conceptualized 
this as successful completion of courses that are part of a 
college preparatory course of study. The availability of data in 
Ohio has changed the way in which we defined this outcome. 
ODE data provided information on course enrollments and 
scores on subject-specific End of Course (EOC) tests. The 
data did not include information about the grade a student 
received or whether a student passed a course. Thus, instead 
of conceptualizing a student as on-track for graduation based 
on passing ninth grade math and English coursework, we 
used data from EOC assessments to determine whether a 
student was on-track for graduation. 


Under a state policy that began with the high school 
graduating class of 2018, one path for high school graduation 
involves earning points based on achievement levels on seven 
specific EOC tests.> Students can earn between 1 and 5 EOC 
graduation points for each subject, determined by their score 
on each test. To earn their high school diploma through this 
pathway, students must earn 18 total points on EOC tests 
across the seven subjects. In addition, students must earn at 
least four points in math (across two subjects), four points in 
English (across two subjects), and six points in science and 
social studies (across three subjects). 


For this study, we defined a student’s status as on-track for 


graduation in 9th grade as follows. By the end of 9th grade, 
students should have earned at least five total graduation 


‘http://education.ohio.gov/Topics/Ohio-s-Graduation-Requirements/18-Points-on-StateTests 


points, with at least two points in a first-level math course 
(Algebra I or Integrated Math I) and at least two points in 

a first-level English course (English I). Each student in the 
ninth grade analytic sample received a value of 1 if they met 
these criteria and a 0 otherwise. Because some students took 
their first-level high school courses and the accompanying 
EOC in the eighth grade, the tests that contributed to on- 
track status may have been taken in either the 2016-17 or 
2017-18 school years for the 2017-18 analysis in either the 
2017-18 or 2018-19 school years for the 2018-19 analysis. 


Analytic Sample. In identifying students who should be 
included in the ninth grade analytic sample, we did not want 
to include students who had entered the high school mid- 
way through their ninth grade year because these students 
would not have had the full benefit of the intervention. The 
administrative data from ODE did not include enrollment 
dates for students. However, the data included each student’s 
Average Daily Membership (ADM), which designated the 
proportion of the school year students were enrolled at 

each Ohio public school. In the outcome analytic sample, 

we included all ninth grade students who attended one of 
the 48 sample schools and who had an ADM of .95 or higher. 
Students who dropped out of one of the sample schools 
during the year in which outcomes were measured were also 
included in the sample, regardless of their ADM. Students 
with an ADM of less than .95 (indicating that they had 
moved schools during the year) were excluded. For the main 
confirmatory analysis, we combined data from Years 2 and 

3 (2017-18 and 2018-19) to create a pooled sample. When 
combining the data, we included student-level observations 
that met the sample criteria for each year and dummy-coded 
the project year (2017-18 as 0 and 2018-19 as 1) as a covariate 
in each model. 


Baseline Equivalence. As noted above, we calculated baseline 
equivalence for the students in the analytic sample. As Table 
III-5 (page 19) shows, the sample of students was equivalent 

when we looked at the relevant demographic characteristics 
and achievement measures. 
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Table III-5. Student-Level Baseline Equivalence for the Ninth Grade On-Track Sample 


Adj. Unadj. 
Treat Comp Treatment Comparison Difference Effect 
WETa t=} 2) (=) N N Mean Mean (Beta) yy 4- (J) P)) 
Pct Economically 7,749 14,481 
: 651 654 -.003 -.01 
Disadvantaged (16 sch) (32 sch) 
Pct 
7,749 14,481 
Underrepresented 513 495 .018 04 
ais (16 sch) (32 sch) 
All Students Race/Ethnicity 
Pooled 
ELA Baseline 7,749 14,481 -.268 -.245 oe a 
Z-Score (16 sch) (32 sch) (SD=.963) (SD=.980) : , 
Math Baseline 7,749 14,481 -.164 -123 aa ee 
Z-Score (16 sch) (32 sch) (SD=1.02) (SD=1.11) : : 
Pct Economically 3,919 7,328 
. 648 649 -.001 .0O 
Disadvantaged (16 sch) (32 sch) 
Pct 
Underrepresented ae eae 512 493 019 05 
All Students p ~~ (16 sch (32 sch) ¢ ° i 
Year 2 Race/Ethnicity 
(2017-18) 
ELA Baseline 3,919 7,328 -.323 -.290 ons a 
Z-Score (16 sch) (32 sch) (SD=.968) (SD=1.00) : : 
Math Baseline 3,919 7,328 -.166 -.151 aie a2 
Z-Score (16 sch) (32 sch) (SD=.992) (SD=1.09) i : 
Pct Economically 3,830 7,153 
: .657 .660 -.003 -.01 
Disadvantaged (16 sch) (32 sch) 
Pct 
3,830 7,153 
Underrepresented 513 496 .017 04 
All Students (16 sch) (32 sch) 
Race/Ethnicity 
Year 3 
(2018-19) ; 
ELA Baseline 3,830 7,153 -.213 -.199 ae ae 
Z-Score (16 sch) (32 sch) (SD=.954) (SD=.953) : : 
Math Baseline 3,830 7,153 -.162 -.095 oer oF 
Z-Score (16 sch) (32 sch) (SD=1.06) (SD=1.14) : : 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 
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‘We repeated this procedure to calculate student-level (not reported here) indicated that there was equivalence for 
baseline equivalence for the sub-groups. Table III-6 shows each of the individual year samples as well. For this outcome, 
that all measures met standards of equivalence except we also added another sub-group that included students who 
for the race/ethnicity measure for the non-economically entered high school without any graduation points; baseline 
disadvantaged sub-group. Care should be taken in equivalence for this sample is shown in the table. 


interpreting results for that subgroup. Separate analyses 


Table III-6. Student-Level Baseline Equivalence for the Ninth Grade On-Track Sample—Pooled Year Subgroups 


Adj. Unadj. 
Treat Comp Treatment | Comparison | Difference Effect 
WEVar-Ve) (=) N N Mean Mean (=f) Ty 4-m (1 P)) 


Pct Economically 5,074 9,474 
i 1.000 1.000 .000 .0O 
Disadvantaged (16 sch) (32 sch) 
i Pct Und ted 5,074 9,474 
Ecanormically ct Un pilepreseh e 700 653 047 B 
Disadvantaged Race/Ethnicity (16 sch) (32 sch) 
seis ee ELA Baseline Z-Score ae eae ieee baa 004 01 
Peele (16 sch) (32 sch (SD=.920)  (SD=.901) , 
Math Baseline 5,074 9,474 -445 -.466 021 02 
Z-Score (16 sch) (32 sch) (SD=.958) (SD=.966) : : 
Pct Economically 2,675 4,973 
. .000 .000 .000 .0O 
Disadvantaged (12 sch) (18 sch) 
Pct Underrepresented 2,675 4,973 
3 277 191 .086 29 
Not EDS Race/Ethnicity (12 sch) (18 sch) 
Students Pooled 2,675 4,973 151 280 
ELA Baseline Z-Score ‘ , -129 -14 
(12 sch) (18 sch) (SD=.913) (SD=.907) 
Math Baseline 2,675 4,973 .297 533 236 oI 
Z-Score (12 sch) (18 sch) (SD=.997) (SD=1.08) : : 
Pct Economically 3,684 7,166 
. .868 863 .005 03 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 3,684 7,166 1000 1.000 000 00 
Underrepresented Race/Ethnicity (16 sch) (32 sch) : : : , 
Race/Ethnicity 
: 3,684 7,166 -.619 -.621 
Pooled ELA Baseline Z-Score 002 .00 
(16 sch) (32 sch) (SD=.899) (SD=.863) 
Math Baseline 3,684 7,166 -.437 -.587 150 16 
Z-Score (16 sch) (32 sch) (SD=.952) (SD=.903) : : 
Pct Economically 4,065 7,299 
: 435 448 -.013 -.03 
Disadvantaged (16 sch) (29 sch) 
Pct Underrepresented 4,065 7,299 000 000 000 00 
Not Under- Race/Ethnicity (16 sch) (29 sch) , : : ‘ 
represented Race/ 
a 4,065 7,299 -.003 125 
Ethnicity Pooled ELA Baseline Z-Score -128 -13 
(16 sch) (29 sch) (SD=.963) (SD=.946) 
Math Baseline 4,065 7,299 159 332 173 16 
Z-Score (16 sch) (29 sch) (SD=1.02) (SD=1.N1) : : 
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Table III-6. Student-Level Baseline Equivalence for the Ninth Grade On-Track Sample—Pooled Year Subgroups 
(continued) 


Variable N 
Pct Economically 6,186 
Disadvantaged (16 sch) 
Pct Underrepresented 6,186 
Students with No Race/Ethnicity (16 sch) 
Pre-HS Graduation 
. : 6,186 
Points ELA Baseline Z-Score 
(16 sch) 
Math Baseline 6,186 
Z-Score (16 sch) 


Adj. Unadj. 
Comp Treatment | Comparison | Difference Effect 
N Mean Mean [(=%=4 =) TP 4-¥ f=] P)) 
11,649 
.701 -706 -.005 -.02 
(32 sch) 
11,649 
563 543 .020 05 
(32 sch) 
11,649 -.625 -.594 
-.031 -.04 
(32 sch) (SD=.841) (SD=.865) 
11,649 -.249 -.245 
-.004 -.01 
(32 sch) (SD=.877) (SD=.955) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 


I11.1.2.2. Tenth Grade On-Track in 2018-19 

Outcome 2 measured the extent to which students were 
on-track for high school graduation in sample schools by the 
end of their tenth grade year. Like Outcome 1, we determined 
on-track status using graduation points students earned on 
End of Course (EOC) tests. 


Outcome Variable. We defined a student’s status as on-track 

in tenth grade in a similar manner to the ninth grade on-track 
outcome. By the end of tenth grade, students should have 
taken six of the seven courses with EOC tests required for 
graduation, two tests in math, two tests in English, and two 
tests in science and social studies. We defined students who 
earned at least 15 total points by the end of tenth grade as on- 
track to reach the 18 points needed for high school graduation. 
In addition, students needed to have earned at least four points 
respectively in each subject category of English, math, and 
science/social studies. Students meeting the total graduation 
point threshold for each subject received a value of 1. Students 
not meeting one or more of these criteria received a 0. For the 
assessment of tenth grade on-track status in 2018-19, students 
could have taken tests contributing to their graduation points 
in all years up to and including 2018-19. Thus, we looked at the 
cumulative assessment file from 2015-16 to 2018-19 to account 
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for all tests that students took through the end of their tenth 
grade year 


Analytic Sample. We used the same criteria to select students 
for the tenth grade sample in Outcome 2 as the ninth grade 
sample in Outcome 1. We included all tenth grade students 
who attended one of the 48 sample schools and who had an 
ADM of .95 or higher. Students who dropped out of one of 

the sample schools during the year in which outcomes were 
measured were also included in the sample, regardless of their 
ADM. Students with an ADM of less than .95 (indicating that 
they had moved schools during the year) were excluded. 


Baseline Equivalence for Analytic Sample. As with the 
ninth grade sample, we calculated baseline equivalence 

for the students in this analytic sample. As Table ITI-7 
(page 22) shows, the student sample was equivalent for the 
measures we examined. 


Table ITI-8 (page 23) shows the equivalence for the specific 
sub-groups of interest. All measures met standards of 
equivalence except for the race/ethnicity measure for the 
non-economically disadvantaged sub-group. Care should be 
taken in interpreting results for that subgroup. 
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Table III-7. Baseline Equivalence for the Year 3 Tenth Grade Analytic Sample 


Adj. Unadj. 
Treatment Comparison Difference Effect Size 
WET at] 2) (=) Treat N Comp N Mean Mean (2X21) (JP) 
Pct Economically 3,422 6,582 
. .618 .616 .002 01 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 3,422 6,582 
Race/Ethnicit ey ——— 502 471 031 .08 
All’ Students ace, nicity (16 sch) (32 sch) 
2018-19 : 3,422 6,582 -.230 -.203 
ELA Baseline Z-Score -.027 -.03 
(16 sch) (32 sch) (SD=.946) (SD=.989) 
: 3,422 6,582 -.054 -.055 
Math Baseline Z-Score .001 .0O 
(16 sch) (32 sch) (SD=.958) (SD=1.08) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 


111.1.2.3. Dropping Out of School in 2018-19 

For Outcome 3, we explored the impact of the CCRE program 
on students dropping out of school. We defined dropouts 
using withdrawal codes in the ODE data. 


Outcome Variable. We defined the dropout outcome as ODE 
records with withdrawal codes of 71-79, described in Table 
IIL-9 (page 24). Ifa student appeared in the list of withdrawal 
codes in this range, they were coded as a1 for dropping out. 
All other enrolled students meeting the ADM threshold 

in each program school were coded as 0 for the dropout 
outcome. 


Analytic Sample. We built our student-level analytic sample 
from the enrollment data for the 2018-19 school year, 
including students in grades 9-12 in the 48 sample schools. 
For students with records at multiple schools, we retained 
the record for which they recorded the maximum ADM. We 
considered a student to be enrolled in a school for the year if 
they had an ADM value of .95 or greater or dropped out of a 
sample school in 2018-19. 


Missing baseline test scores were a concern for students who 
dropped out because these students were more likely to not 
have baseline test scores recorded in the ODE data. Our full 


impact model for the confirmatory analysis did not include 
baseline test scores due to losing many of the students who 
dropped out from the analytic sample. Thus, we did not 
exclude students with missing baseline test covariates, as 
we did for the other analyses. It is important to note that 
this is acceptable given that we have already demonstrated 
school-level baseline equivalence on dropout rates (see 
Table III-4, page 17) using the same approach to create 

the baseline measure as we used for the outcome measure. 
The representativeness of both the baseline measure and 
the outcome measure samples can be found in Table B-3 in 
Appendix B. 


To ensure this decision did not alter our interpretation of 
the results, we conducted a follow-up sensitivity analysis for 
students with non-missing baseline test scores and assessed 
baseline equivalence for that analytic sample as well. We also 
used the sample with non-missing test scores to conduct all 
subgroup analyses. 


There was also a possibility that a student dropped out of 
a sample school prior to the 2018-19 school year (but was 
recorded as a dropout for the 2018-19 school year) and 

did not ever get recorded in the year’s enrollment file. We 
added 355 records to the analytic sample that signaled this 
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Table III-8. Baseline Equivalence for the Year 3 Tenth Grade Subgroup Analytic Samples 


EDS Students 
2018-19 


Not EDS Students 
2018-19 


Underrepresented 
Race/ 
Ethnicity 2018-19 


Not Underrepresented 
Race/ 
Ethnicity 2018-19 


Variable 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Treat N 


2,]11 
(16 sch) 


2,111 
(16 sch) 


2111 
(16 sch) 


2,111 
(16 sch 


1,311 
(11 sch) 


1,311 
(11 sch) 


1,311 
(11 sch) 


1,371 
(11 sch) 


1,584 
(16 sch) 


1,584 
(16 sch) 


1,584 
(16 sch) 


1,584 
(16 sch) 


1,838 
(16 sch) 


1,838 
(16 sch) 


1,838 
(16 sch) 


1,838 
(16 sch) 


Comp N 


4,056 
(32 sch) 


4,056 
(32 sch) 


4,056 
(32 sch) 


4,056 
(32 sch) 


2,526 
(19 sch 


2,526 
(19 sch 


2,526 
(19 sch) 


2,526 
(19 sch 


3,099 
(32 sch) 


3,099 
(32 sch 


3,099 
(32 sch 


3,099 
(32 sch 


3,477 
(29 sch) 


3,477 
(29 sch) 


3,477 
(29 sch) 


3,477 
(29 sch) 


Adj. 


Treatment 
Mean 


1.000 


693 
-478 
(SD=.904) 
“sie 
(SD=.908) 


.000 


283 
127 
(SD=.907) 
299 
(SD=.940) 


.836 


1.000 
-545 
(SD=.881) 
-415 
(SD=.891) 


.406 


.000 
-.002 
(SD=.944) 


.200 
(SD=.946) 


Unadj. 
Comparison 
Mean 


1.000 


642 
-496 
(SD=.913) 
-396 
(SD=.948) 


.00O0 


197 
268 
(SD=.922) 
492 
(SD=1.05) 


.840 


1.000 
-584 
(SD=.867) 
-518 
(SD=.879) 


417 


.000 
136 
(SD=.968) 


356 
(SD=1.07) 


Difference | Effect Size 
[(=f=3 8) (>) 
.000 .0O 
.051 14 
.018 .O2 
.084 .09 
.000 .0O 
.086 52 
-141 -15 
-.193 -.21 
-.004 -.02 
.000 .0O 
.039 .05 
103 1 
-.O1] -.03 
.00O0 .0O 
-.138 -.15 
-.156 -17 
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Table III-9. ODE Withdrawal Codes Used in Dropout Definition 


Withdrawal Code Withdrawal reason& 


7I Withdrew due to truancy/nonattendance 

72 Pursued employment/work permit 

73 Over 18 years of age 

74 Moved, not known to be continuing 

75 Student completed course requirements but did not pass statewide assessments 
required for graduation 

76 Non-attendance according to the 72-hour rule 

79 No longer eligible to be enrolled in district 


scenario—a dropout record attached to a sample school in 
2018-19 with no accompanying enrollment record. For these 
students, we used student demographic covariates from a 
prior school year for our models. 


Baseline Equivalence. Because our main analysis did not 
include baseline test scores as covariates, we demonstrated 
equivalence at the school level for this outcome (see 

Tables ITI-3 and III-4). As an additional check, we also 
assessed baseline equivalence on this sample at the student 
level on the percentage economically disadvantaged, 
underrepresented race/ethnicity, and missing one or both 
baseline achievement measures, summarized in Table ITI-10 
(page 25). 


‘We also assessed measures of baseline equivalence for the 


students in our sensitivity analysis who had no missing data. 


Table ITI-11 (page 26) shows the groups were equivalent 

on the demographic characteristics, on the percentage of 
students missing achievement data and on achievement for 
those students with non-missing data. These analyses were 
replicated with the various sub-groups with all differences 
falling below the 0.25 standard deviation threshold 
recommended by the WWC Standards. 


111.1.2.4. College Course Enrollment in 2018-19 
Outcome 4 addressed the impact of the CCRE program on 
college course enrollment. We defined the college course 
enrollment outcome in three ways: students taking a) one 
or more of either College Credit Plus (CCP) or Advanced 
Placement (AP) courses, b) one or more CCP courses, or c) 
one or more AP courses. 


Outcome Variable. For Outcome 4, we wanted to determine 
how many students had one or more records corresponding to 
a college-level course, defined as either a College Credit Plus 
(CCP) or Advanced Placement (AP) course. This information 
came from the ODE course information file with information 
about all courses taught in Ohio schools. We filtered it to 
include only those courses with curriculum code values of 
PS (Postsecondary, which applied to CCP courses) and AP. 
We then created two dichotomous variables in the student 
analytic sample that indicated whether a student took at 
least one CCP course and at least one AP course. From these 
values we calculated a third dichotomous variable indicating 
whether a student took at least one of either of these types 
of college-level courses. These three measures serve as 

the student-level outcome variables for the college course 
enrollment outcome. 


SEMIS Manual 2.4, Student Standing (FS) Record (http://education.ohio.gov/getattachment/Topics/Data/EMIS/EMIS-Documentation/Current-EMIS-Manual/2-4-Student-Standing-FS-Record-v10-2.pdf.aspx?lang=en-US) 
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Table III-10. Baseline Equivalence for the Year 3 Dropout Analytic Sample 


WEVar-ye) (=) Treat N 
Pct Economically 14,875 
Disadvantaged (16 sch) 
Pct Underrepresented 14,875 
Race/Ethnicity (16 sch) 
All 
Students Pct Missing Either ELA or 14,875 
2018-19 Math Baseline Score (16 sch) 
ELA Baseline Z-Score (Non- 13,382 
Missing) (16 sch) 
Math Baseline Z-Score 13,382 
(Non-Missing) (16 sch) 


Adj. Unadj. 
Treatment | Comparison | Difference 
Comp N Mean Mean (2X2) 
27,845 
634 632 .002 01 
(32 sch) 
27,845 
.508 488 .020 05 
(32 sch) 
27,845 
100 .081 O19 14 
(32 sch) 
25,593 -.201 -.193 
-.008 -.01 
(32 sch) (SD=.974) (SD=.996) 
25,593 -.106 -094 
-.012 -.01 
(32 sch) (SD=.929) (SD=1.03) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 


Analytic Sample. We built our student-level analytic sample 
from the enrollment data for the 2018-19 school year, 
including students in grades 10-12 in the 48 sample schools. 
Our original plan was to include students in grades 9-12, 
however, starting in the 2019-20 school year, one of the 
treatment districts moved all ninth-grade students in the 
district to a new school. Because this was a new school in 
2019-20, including ninth-grade students in the analysis 
with Year 4 outcomes would be problematic due to missing 
school-level baseline covariates. To deal with this issue, we 
decided to change the analytic sample to students in grades 
10-12 so we could include both Year 3 and Year 4 outcomes in 
our analysis. 


For students with records at multiple schools, we retained 
the record for which they recorded the maximum ADM. 
Students in the sample with missing data for any of the 
model covariates were also removed from the sample using 
listwise deletion. We also considered a student to be enrolled 
in a school for the year if they had an ADM value of .50 or 


greater or dropped out of a sample school in 2018-19. We 
chose a different ADM threshold for this outcome because 
many eleventh and twelfth grade students in Ohio were 
concurrently enrolled part-time in their home high school 
and part-time in other programs (such as a career center or 
other high school offering specialized coursework). To ensure 
we captured these students in the sample, we included all 
students who were enrolled in a sample school at least half of 


their time. 


Baseline Equivalence. Our analyses of baseline equivalence for 
the students in the analytic sample show that the sample was 
equivalent (Table III-12, page 27). 


Table ITI-13 (pages 28-29)) shows the baseline equivalence 
for the students in the sub-groups for the pooled sample. As 
the table shows, no differences between the two sub-groups 
exceeded the .25 standard deviation threshold; thus, the 
groups can be considered equivalent. 
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Table III-11. Baseline Equivalence for the Year 3 Dropout Analytic Sample Subgroups 


sere | te 


All Students with 
Non-Missing Baseline 
Test Scores 2018-19 


EDS Students with Non- 
Missing Baseline Test 
Scores 2018-19 


Not EDS Students with 
Non-Missing Baseline 
Test Scores 2018-19 


Under-represented 
Race/Ethnicity 
with Non-Missing 
Baseline 

Test Scores 2018-19 


Not 
Under-represented 
Race/ Ethnicity with 
Non-Missing Baseline 
Test Scores 2018-19 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline 
Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Treat N 


13,382 
(16 sch) 


13,382 
(16 sch) 


13,382 
(16 sch) 


13,382 
(16 sch) 


8,205 
(16 sch) 


8,205 
(16 sch) 


8,205 
(16 sch) 


8,205 
(16 sch) 


5,177 
(12 sch) 


5,177 
(12 sch) 


5,177 
(12 sch) 


5,177 
(12 sch) 


6,163 
(16 sch) 


6,163 
(16 sch) 


6,163 
(16 sch) 


6,163 
(16 sch) 


7,219 
(16 sch) 


7,219 
(16 sch) 


7,219 
(16 sch) 


7,219 
(16 sch) 


Comp N 


25,593 
(32 sch) 


25,593 
(32 sch) 


25,593 
(32 sch) 


25,593 
(32 sch) 


16,054 
(32 sch) 


16,054 
(32 sch) 


16,054 
(32 sch) 


16,054 
(32 sch) 


9,539 
(20 sch) 


9,539 
(20 sch) 


9,539 
(20 sch) 


9,539 
(20 sch) 


12,353 
(32 sch) 


12,353 
(32 sch) 


12,353 
(32 sch) 


12,353 
(32 sch) 


13,240 
(29 sch) 


13,240 
(29 sch) 


13,240 
(29 sch) 


13,240 
(29 sch) 


Treatment 
Mean 


621 


503 
-197 
(SD=.971) 
-102 
(SD=.928) 


1.000 


704 
-451 
(SD=.931) 
-365 
(SD=.916) 


.000 


242 
159 
(SD=.938) 
.260 
(SD=.861) 


.853 


1.000 
-539 
(SD=.918) 
-483 
(SD=.902) 


385 


.000 


.070 
(SD=.967) 


219 
(SD=.890) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 
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Comparison 
Mean 


627 


483 


-192 
(SD=.995) 
-.091 
(SD=1.02) 


1.000 


654 
475 
(SD=.928) 
-417 
(SD=.919) 


.000 


195 
282 
(SD=.924) 
452 
(SD=.962) 


.850 


1.000 
-.567 
(SD=.885) 
-526 
(SD=.872) 


420 


.000 


156 
(SD=.967) 


3n 
(SD=.992) 


Difference 
(2X21) 


-.006 


.020 


-.005 


-.O11 


.000 


.050 


.024 


.052 


.00O 


047 


-.123 


-.192 


.003 


.00O 


.028 


.043 


-.035 


.000 


-.086 


-.092 


Effect Size 
12) 


-.02 


.O5 


-.01 


-.01 


.0O 


4 


.03 


.O5 


.0O 


17 


-.13 


-.21 


01 


.0O 


.03 


.O5 


-.09 


.0O 


-.09 


-.10 
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Table III-12. Baseline Equivalence for the College Coursetaking Analytic Samples 


Adj. Unadj. Difference | Effect Size 


[(={=1¢-)) (J) 


WETat-] 2) (=) Treat N Comp N Treatment | Comparison 
Mean Mean 


Pct Economically 20,929 41,628 
: .627 593 .034 .09 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 20,929 41,628 
we , : 509 483 .026 .O6 
All Grades 10-12 Base/ Erhmichy (16 sch) (32 sch) 
Students Pooled 
ELA Baseline Z-Score 20,929 41,628 -.192 -.188 seu és 
(16 sch) (32 sch) (SD=.971) (SD=.985) 
Math Baseline Z-Score 20,929 41,628 -.082 -.089 oe ei 
(16 sch) (32 sch) (SD=.967) (SD=.967) 
Pct Economically 10,192 20,524 
. .610 .617 -.007 -.02 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 10,192 20,524 
. .498 477 .021 .O5 
Race/Ethnicity (16 sch) (32 sch) 
All Grades 10-12 
Students 2018-19 
ics 10,192 20,524 “55 -170 
ELA Baseline Z-Score O15 02 
(16 sch) (32 sch) (SD=.957) (SD=.982) 
. 10,192 20,524 -.058 -.072 
Math Baseline Z-Score O14 02 
(16 sch) (32 sch) (SD=.917) (SD=1.02) 
Pct Economically 10,737 21,104 
: 643 570 .073 19 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 10,737 21,104. 
Be: 520 .490 .030 .O7 
Race/Ethnicity (16 sch) (32 sch) 
All Grades 10-12 
Students 2019-20 
. 10,737 21,104 -.226 -.206 
ELA Baseline Z-Score -.020 -.02 
(16 sch) (32 sch) (SD=.983) (SD=.988) 
: 10,737 21,104 -.104 -.106 
Math Baseline Z-Score .0O1 .0O 
(16 sch) (32 sch) (SD=.920) (SD=1.02) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 


serve.uncg.edu 


28 | CREATING A FUTURE-ORIENTED CULTURE IN HIGH SCHOOLS 


Table III-13. Baseline Equivalence for the Pooled College Coursetaking Subgroup Analytic Samples 


EDS 
Grades 10-12 
Students Pooled 


Not EDS 
Grades 10-12 
Students Pooled 


Underrepresented 
Race/Ethnicity 
(Grades 10-12) 


Not 
Underrepresented 
Race/Ethnicity 
(Grades 10-12) 


WEVat-] 0) {= 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Treat N 


12,758 
(16 sch) 


12,758 
(16 sch) 


12,758 
(16 sch) 


12,758 
(16 sch) 


8,171 
(12 sch) 


8,171 
(12 sch) 


8,171 
(12 sch) 


8,171 
(12 sch) 


9,928 
(16 sch) 


9,928 
(16 sch) 


9,928 
(16 sch) 


9,928 
(16 sch) 


11,001 
(16 sch) 


11,001 
(16 sch) 


11,001 
(16 sch) 


11,001 
(16 sch) 


Comp N 


24,685 
(32 sch) 


24,685 
(32 sch) 


24,685 
(32 sch) 


24,685 
(32 sch) 


15,886 
(21 sch) 


15,886 
(21 sch) 


15,886 
(21 sch) 


15,886 
(21 sch) 


20,126 
(32 sch) 


20,126 
(32 sch) 


20,126 
(32 sch) 


20,126 
(32 sch) 


21,458 
(29 sch) 


21,458 
(29 sch) 


21,458 
(29 sch) 


21,458 
(29 sch) 


Adj. 
Treatment 
Mean 


1.000 


695 


-432 
(SD=.930) 


-.329 
(SD=.908) 


.00O0 


.200 


207 
(SD=.937) 


336 
(SD=.859) 


832 


1.000 


-515 
(SD=.916) 


-449 
(SD=.890) 


416 


000 
072 
(SD=.961) 


222 
(SD=.879) 


Comparison | Difference 


1.000 


.653 


-459 
(SD=.926) 


-402 
(SD=.922) 


.00O 


.237 


205 
(SD=.935) 


366 
(SD=.974) 


.800 


1.000 


-555 
(SD=.871) 


-518 
(SD=.869) 


399 


000 
155 
(SD=.961) 


313 
(SD=.979) 


.000 


042 


.026 


073 


.000 


-.037 


.0O1 


-.030 


.032 


.00O0 


.040 


.069 


017 


.000 


-.083 


-.090 


Effect Size 
}p)) 


.0O 


12 


.03 


.08 


.0O 


-.13 


.0O 


-.03 


13 


.0O 


.OS 


.08 


04 


.0O 


-.09 


-.10 
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Table III-13. Baseline Equivalence for the Pooled College Coursetaking Subgroup Analytic Samples (continued) 


Variable Treat N 
Pct Economically 13,216 
Disadvantaged (16 sch) 
Pct Underrepresented 13,216 
Race/Ethnicity (16 sch) 
Grades 11-12 Pooled 
. 13,216 
ELA Baseline Z-Score 
(16 sch) 
: 13,216 
Math Baseline Z-Score 
(16 sch) 
Pct Economically 7,713 
Disadvantaged (16 sch) 
Pct Underrepresented 7,713 
Race/Ethnicity (16 sch) 
Grade 10 Pooled 
: 7,713 
ELA Baseline Z-Score 
(16 sch) 
: 7,713 
Math Baseline Z-Score 
(16 sch) 


Adj. 
Treatment | Comparison | Difference | Effect Size 
Comp N Mean (JP) 
26,638 
612 582 .030 .08 
(32 sch) 
26,638 
.506 483 .023 05 
(32 sch) 
26,638 -149 -.154 
.005 01 
(32 sch) (SD=.970) (SD=.983) 
26,638 -.050 -.060 
.O10 01 
(32 sch) (SD=.924) (SD=1.02) 
14,990 
.653 612 042 1 
(32 sch) 
14,990 
515 483 .031 .08 
(32 sch) 
14,990 -.266 -.250 
-.016 -.02 
(32 sch) (SD=.969) (SD=.986) 
14,990 -.136 -.140 
.004 .0O 
(32 sch) (SD=.906) (SD=1.02) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox’s index for dichotomous variables. 


111.1.2.5. Average College Credits Earned by 
Graduates in 2018-19 


Outcome 5 examined the impact of the CCRE program on 
students earning college credit. For our analysis, within the 
treatment and comparison schools, students could receive 
college credit either by successfully completing a College 
Credit Plus course or by scoring a 3 or higher on an Advanced 
Placement exam. The confirmatory analysis focused on the 
impact of the average combined number of college credits 
earned from CCP and AP for students graduating from high 
school in 2018-19. To better understand patterns of students 
earning college credits we also ran several sub-analyses on 
the following outcomes: 1) the average number of dual credits 
earned, 2) the average number of AP credits earned, 3) 
whether a student graduated with any dual credit, 4) whether 


a student graduated with three or more dual credits, and 5) 
whether a student graduated with any AP credit equivalent. 


Outcome Variable. The number of dual credits earned and 
student scores on AP exams come from two separate ODE 
files and require different procedures for calculating the total 
college credits. The confirmatory outcome variable is the sum 
of dual credits and equivalent AP credits; we discuss each 
calculation separately in the following paragraphs. 


e Dual credits earned (CCP). The graduation core file 
contained multiple records for each student, with a 
row for each subject area (such as English or CTE). 
The file recorded the number of dual credits a student 
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earned in each subject area. We calculated each 
student's total dual credits earned by taking the sum 
of the dual credits earned across all subject areas. 

We also used these values to calculate dichotomous 
variables indicating whether a student earned any dual 
credit and whether a student earned three or more 
dual credits (a threshold used in Ohio accountability 
Prepared for Success measures’). 


e AP Credit Equivalent. To determine AP credit 
equivalents, we used AP test data from ODE. Each 
record in the file corresponded to an AP test subject 
and score taken by students during high school. ODE 
did not report the college credit equivalents for AP 
test scores. We used a crosswalk of AP college credit 
equivalents to determine the number of college 
credits students earn for different AP scores at Ohio 
University. This crosswalk is available upon request. 


Analytic Sample. ODE does not track the number of dual 
credits students earned each year of high school. Schools 
and districts are only required to report the number of 
dual credits that students earned throughout high school 
at the time of student graduation. As such, our analytic 
sample included twelfth grade students who appeared in 
the graduation core file from the same 48 treatment and 
comparison schools as the other analyses. Because college 
credits earned was a cumulative outcome, we did not place 
any ADM restrictions on the sample, considering a student 
as enrolled in the sample school that was attached to their 
graduation record. 


Baseline Equivalence. As shown in Table III-14 (page 33), the 


samples were equivalent at baseline. 


As shown in Table ITI-15 (page 33), baseline equivalence 
existed at the student level for the pooled sample; we also 
found equivalence for each of the specific years (not shown). 


11.1.3. Analytic Approach 


We used hierarchical linear modeling (HLM) (Raudenbush 
& Bryk, 1992) as the general analytic framework to account 
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for the nested structure of the data. For all analyses, students 
were nested within schools. In general, these models seek to 
answer the question, “Is there an overall treatment effect of 
the intervention on relevant student outcomes for schools 
that implement the model relative to their comparison school 
counterparts?” Consequently, our models included a fixed 
treatment effect at level 2, which was the primary effect of 
interest. The models also included the baseline measures that 
are designated as the variables on which baseline equivalence 
is established. 


We adjusted for multiple comparisons within each of the 
three domains as appropriate. Outcomes 1 & 2 were subject 
to adjustment because they fell within the same domain. 
Outcome 3 was the only outcome in its domain, so it was 
not adjusted for multiple comparisons. Finally, Outcomes 4 
& 5 were also within the same domain and were subject to 
adjustment. We used significance tests that account for the 
potential false discovery rate (Benjamini & Hochberg, 1995) 
for outcomes within the same domain. We did not perform 
adjustments for exploratory analyses. 


Model Specifications. The following model includes the 
general specifications below for all outcomes. 


Level 1 (student level) 


p 
Viz = Po + > 4yCOV,, +2é; 


p=l 


where: 


= outcome of interest for student i in school j; 


ct: 


COV, , = p-th student-level covariate included in 
the final model; 


2, j  =adjusted mean outcome of interest for school 
j controlling for differences in student-level 


covariates; 


B yy = the association between the p-th student-level 
covariate and outcome of interest; and 
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e, = random effect of student i in school j assumed to 


be distributed with a mean of zero and variance of 
2 


om 


e 


Level 2 (school level) 


K 
Boj = Yoo + Yo17j + > Yoce+2)X}° + Uo; 


By = 7 po 

where: 

T, = the indicator variable showing whether school j 
was a treatment (1) or matched comparison 
school (0); 

xX : = k-th (k=1,2....,K) school-level measure used in 


the matching process; 


Yo0 = adjusted mean of the outcome of interest in 
the comparison group; 


Yor = overall fixed-treatment effect adjusted for 


the covariates; 


Ypo _=pooled within-school regression coefficient for 
student-level covariate p; and 


= random effect of school j, assumed to be distributed 
with a mean of zero and variance of ; note that 
this term is also assumed to be independent of the 


student-level error term, ei: 


‘We adapted this generic model to each outcome, and we 
estimated using a two-tailed significance test at the p < .05 
significance level. The coefficient represents the overall 
treatment effect in each model. 


Thttp://education.ohio.gov/Topics/Data/Report-Card-Resources/Prepared-for-Success-Component 


The variables included in the impact model are listed in 
Table III-16 (page 34). 


111.1.3.1. Subgroup Analyses 

For each outcome, we repeated the analytic model 
described above for four different demographic sub- 
groups: 1) economically disadvantaged students, 2) not 
economically disadvantaged students, 3) students from 
underrepresented race/ethnic groups, and 4) students from 
not underrepresented race/ethnic groups. For the college 
coursetaking outcomes, we also ran the analyses for three 
different combinations of grade levels: 1) Grades 10-12 (the 
primary analyses), 2) Grades 11-12, and 3) Grade 10. For 
each subgroup analysis, we replaced the baseline school- 
level variables with measures specific to the subgroup. For 
example, all analyses of economically disadvantaged students 
included baseline covariates specific to that group (e.g., 

the school-level percentage of economically disadvantaged 
students on track in ninth grade). 


111.1.3.2. Missing Data 

The administrative data from Ohio Department of Education 
had little missingness for the variables included in the 
analyses. As such, we decided to use case-wise deletion 

for students with any missing data. We did not use any 


imputation procedures. 


111.2, MEASURING EARLY COLLEGE DESIGN 
PRINCIPLE IMPLEMENTATION 

This sub-study was intended to examine the impact of 
CCRE on schools, tracking changes over time using a survey 
administered to staff in treatment schools. The survey was 
administered in the fall of 2016, which was intended to serve 
as a baseline, and then again in the falls of 2017, 2018, and 
2019. In the last two years, questions were added about 
perceived impact and sustainability. The content of the 
survey is discussed in more depth below. 
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Table III-14. Baseline Equivalence for the College Credits Earned Analytic Sample 


Adj. Unadj. F F 
Variable Treat N Comp N Treatment Comparison eateliee Se Coie 
Mean Mean 
Pct Economically 6,341 13,039 
. 588 568 .020 05 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 6,341 13,039 
oe 499 478 021 05 
All Grad Core Race/Ethnicity (16 sch) (32 sch) 
Students Pooled 6,341 13,039 123 a2 
ELA Baseline Z-Score -.O11 -.01 
(16 sch) (32 sch) (SD=.943) (SD=.964) 
Math Baseline 6,341 13,039 -.127 -.131 004. 01 
Z-Score (16 sch) (32 sch) (SD=.890) (SD=.967) : : 
Pct Economically 3,060 6,424 
: 574 590 -.016 -.04 
Disadvantaged (16 sch) (32 sch 
Pct Underrepresented 3,060 6,424 
a 503 AT4 .029 .O7 
Ali'Grad Gore Race/Ethnicity (16 sch) (32 sch) 
Students 2018-19 ; 3,060 6,424 “145 -.081 
ELA Baseline Z-Score -.062 -.07 
(16 sch) (32 sch) (SD=.921) (SD=.953) 
Math Baseline 3,060 6,424 -.210 -.166 642 ee 
Z-Score (16 sch) (32 sch) (SD=.847) (SD=.904) , : 
Pct Economically 3,281 6,615 
. .605 547 .058 14 
Disadvantaged (16 sch) (32 sch) 
Pct Underrepresented 3,281 6,615 
.. 496 482 014 .03 
All Grad Core Race/Ethnicity (16 sch) (32 sch) 
Students 2019-20 3,281 6,615 -099 -141 
ELA Baseline Z-Score 042 .04 
(16 sch) (32 sch) (SD=.963) (SD=.973) 
: 3,281 6,615 004 -.048 
Math Baseline Z-Score .052 .06 
(16 sch) (32 sch) (SD=.915) (SD=1.02) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox’s index for dichotomous variables. 
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Table III-15. Baseline Equivalence for the Pooled College Credits Earned Subgroup Analytic Sample 


WEVar-]e) (=) 


Treat N 


Comp N 


Adj. 


Treatment 
Mean 


Unadj. 
Comparison 
Ta) 


Difference 


((=X=€)) 


EDS Grad Core 
Students Pooled 


Not EDS Grad Core 
Students Pooled 


Underrepresented 
Race/Ethnicity 
Grad Core 
Students Pooled 


Not 
Underrepresented 
Race/Ethnicity 
Grad Core 
Students Pooled 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


Pct Economically 
Disadvantaged 


Pct Underrepresented 
Race/Ethnicity 


ELA Baseline Z-Score 


Math Baseline Z-Score 


3,560 
(16 sch) 


3,560 
(16 sch) 


3,560 
(16 sch) 
3,560 
(16 sch) 
2,781 
(12 sch) 


2,781 
(12 sch) 


2,781 
(12 sch) 
2,781 
(12 sch) 
2,915 
(16 sch) 
2,915 
(16 sch) 
2,915 
(16 sch) 
2,915 
(16 sch) 
3,426 
(16 sch) 
3,426 
(16 sch) 
3,426 
(16 sch) 
3,426 
(16 sch) 


7,410 
(32 sch) 


7,410 
(32 sch) 


7,410 
(32 sch) 
7,410 
(32 sch) 
5,307 
(19 sch) 


5,307 
(19 sch) 


5,307 
(19 sch) 
5,307 
(19 sch) 
6,239 
(32 sch) 
6,239 
(32 sch) 
6,239 
(32 sch) 
6,239 
(32 sch) 
6,785 
(29 sch) 
6,785 
(29 sch) 
6,785 
(29 sch) 
6,785 
(29 sch) 


1.000 


-707 


-.365 
(SD=.907) 
-333 
(SD=.874) 


.00O0 


.252 


195 
(SD=.922) 
159 
(SD=.864) 


812 


1.000 


-434 
(SD=.897) 
-457 
(SD=.853) 


380 


.000 


7 
(SD=.936) 
161 
(SD=.860) 


Note: Effect sizes were calculated using Hedges’ g for continuous variables and Cox's index for dichotomous variables. 


1.000 


.668 


23738 
(SD=.917) 
-404 
(SD=.877) 


.000 


.229 


233 
(SD=.915) 
287 
(SD=.939) 


794 


1.000 


-478 
(SD=.856) 
-520 
(SD=.833) 


361 


.000 


224 
(SD=.935) 
274 
(SD=.924) 


.000 


.039 


.008 


O71 


.00O 


.023 


-.043 


-.128 


.018 


.00O0 


044 


.063 


019 


.00O 


-.107 


--113 


.0O 


1 


01 


.08 


.0O 


.08 


-.05 


“14 


.O7 


.0O 


.05 


.O7 


.O5 


.0O 


-11 


-.13 
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Table III-16. Variables Included in Impact Model 


Tod aYoXo) fy =0¥(=) AYE Tat-] 0) (1-3 Student-Level Variables 


% of 9th Grade Students On Track (2015-16) 
% of Students Taking a CCP Course (2015-16) 
% of Students Taking an AP Course (2015-16) 


Mean Number of College Credits Earned (CCP and AP) 
(Graduates in 2015-16) 


Dropout Rate (2015-16) 
% Economically Disadvantaged (2015-16) 
% Underrepresented Race/Ethnicity (2015-16) 


Enrollment in Grades 9-12 in 1000s of 
Students (2015-16) 


11.2.1. Sample 

During the survey window, the seven CCRE district 
coordinators and a data manager at each school oversaw 

the distribution of survey links and other aspects of 
administering the survey. Certified staff members from each 
of the 16 participating schools were provided the opportunity 
to take the survey. Table III-17 summarizes the response 

rate at each school for the four project years. Fifteen of the 
sixteen schools met the expected response rate threshold of 
50%, with an overall response rate of 59% across all program 


Table III-17. Survey Response Rate 


Total Certified 
Staff Eligible (Reported by 


Total Survey 


Economically Disadvantaged (Yes/No) 
Underrepresented Race/Ethnicity (Yes/No) 
Gender 

Disability Status (Yes/No) 

Limited English Proficient Status (Yes/No) 
Math Baseline Z-Score 

ELA Baseline Z-Score 


Student Had Graduation Points Prior to 9th Grade 
(Yes/No)8 


schools. SERVE Center provided school-level summary data 
to each participating site as formative feedback and for use in 
planning. Each school with a response rate over 50% received 
$1,000 for each survey administration. 


111.2.2. Measures 

The staff survey was developed during Fall 2016 utilizing, as a 
starting point, items from other i3 grant projects which were 
aligned to the Early College Design Principles articulated 

in the CCRE project proposal. Scales measuring aspects of 


(o)¥(-1 6-1 | 
Response Rate 


Response Rate Range by Site 


Year sX-¥-4 fe) ao (ah a) District Coordinators) 
Year 1 

836 1,052 
(2016) 
Year 2 

746 1,056 
(2017) 
Year 3 

760 1,055 
(2018) 
Year 4 

619 1,056 
(2019) 


Only included in the models for the on-track analyses, Outcomes 1 and 2. 
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705% 50%-100% 
‘ (All 16 schools met threshold) 
ae 23%-100% 
° (15/16 schools met 50% threshold) 
aes 55%-97% 
(All 16 schools met threshold) 
50%-100% 


59% 
7 (15/16 schools met 50% threshold) 
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each Design Principle from previous projects were adapted to 
reflect the expected early college implementation outcomes 
in CCRE. Staff members from Columbus State and two non- 
program schools in Columbus piloted and vetted the items 
prior to the first administration. A table summarizing the 
scales and their reliability can be found in Appendix C. 


11.2.3. Survey Analysis 

For most questions on the survey, each scale was reported as 

an average of all responses for the school. For questions that 

were presented only to school administrators and counselors, 


an average was calculated for each school. 


For questions posed to administrators and counselors asking 
them to estimate the percentage of students in their school 
to which a certain statement applies (e.g., “enrolled in one or 
more honors courses”), responses were often not consistent 
across respondents from the same school, potentially 
reflecting differing levels of familiarity with students’ 
coursetaking and college preparatory activities. To obtain 

a representative response from each school, the following 
values were considered in the order listed below. If value 1 
existed, it was used as the school’s response. Otherwise, value 


2 was used, and so on. 


1. The overall mode 


2. Ifno overall mode and only one counselor response, the 
counselor’s response 


3. Ifno overall mode and more than one counselor, mode 
among counselor responses 


4. Ifno mode among counselors, the response of the 
counselor who has been at the school the longest 


5.  Ifcounselors did not differ in time at school, the lower 


response value 


6. Ifno responses from a counselor, the response of the 
administrator who has been at the school the longest 


7. Ifadministrators did not differ in time at school, the 
lower response value 
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Generalized Estimating Equations (GEE) analysis was 

used to test for changes in average responses from the 
baseline survey administration (Year 1) to the Year 4 results. 
GEE analysis accounts for the clustering of staff within 
schools and allows us to address issues associated with the 
anonymity of the survey. Many of the surveys were completed 
by the same staff across the program years (which meant 
that the survey responses across years were not independent 
of each other) but, because the survey was anonymous, 
responses could not be linked to the same respondents across 
years. GEE analysis addresses the violation of independence 
due to repeat respondents by employing empirically 
estimated standard errors, which are robust to violations of 
independence. Additionally, like a multi-level model, a GEE 
model accounts for the clustering of survey respondents 
within schools, thereby accounting for the other source of 
dependence between responses. GEE analysis was used for 
all survey questions except those that were presented to 
only administrators and counselors, with the survey year as 
the sole predictor and with responses nested within schools. 
Because the administrator/counselor-only responses were 
aggregated to the school level, nesting was not a concern for 
those specific questions, which were compared from baseline 
to the final year using paired-samples t-tests. 


For the analysis of principal responses between CCRE and 
comparison schools, the evaluation team used a two-sample 
t-test to assess differences between groups. For comparing 
the growth of CCRE schools to comparison schools, the team 
used a repeated-measures analysis of variance (ANOVA) test. 


111.3. CASE STUDIES 
To describe school-level implementation in more detail, the 
evaluation team conducted case studies in six CCRE schools. 


11.3.1. Sample 

For the case studies, the evaluation team worked with the 
project staff to identify six schools in five districts for a 
series of site visits. The sites were purposefully selected 
to represent varying demographics, urbanicity, and 
implementation activities (Table III-18 page 36). 
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Table III.18. Site Visit Schools Demographics 


Percent 
Economically "White, 
Disadvantaged | Non-Hispanic" 
Co) Nod | 
School A 540 53.9% 39.4% 
School B 743 100.0% 42.5% 
School C 1313 77.5% 48.1% 
School D 799 61.9% 78.7% 
School E 955 100.0% 20.7% 
School F 832 98.4% 27.0% 
State 
49.4% 69.6% 
Average 


Source: 2018-19 Ohio School Report Cards? 


11.3.2. Data Sources 

The case studies incorporated data from a variety of sources 
including: 1) site visits with interviews and observations, 2) 
a review of relevant documents, 3) interviews with project 
and district staff, 4) data collected to assess fidelity of 
implementation, and 5) implementation survey data. 


I11.3.2.2. Site Visits 

SERVE Center staff visited each site in fall 2017, 2018, and 
2019; visits included a full day of on-site interviews with 
teachers, counselors, principals, and other key staff for CCRE. 
The visits also included student focus groups and classroom 
observations of teachers participating in CCRE activities. 
Table ITI-19 (page 37) shows the data collected across the six 
schools during site visits. 


The semi-structured interviews included questions focused 
on understanding implementation of the CCRE supports 
as well as school-level implementation of the Early College 
Design Principles. 


*https://reportcard.education.ohio.gov 
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4-Year 


Prepared 
(eie-le [UE Latoya] for Success 
Rate (eT Yo (i 


Achievement 
Tate (=> dl 


Progress 
Grade* 


97.9% F 59.8% (D) Cc 
96.4% F 65.5% (D) A 
83.7% F 53.4% (D) D 
90.7% F 70.7% (C) B 
78.6% F 49.0% (F) S 
78.5% F 52.5% (F) B 


111.3.2.2. Document Review 

The evaluation team collected documents from all schools 
including reports outlining their strategic plans and the 
pathways they were developing. Additionally, we used school- 
level administrative data from the Ohio Department of 
Education to provide implementation context. 


111.3.2.3. Project and District Staff Interviews 

Across the five project years, the evaluation team conducted 
a total of 70 interviews with Columbus State, JFF, ESC, 

and district staff. These interviews focused primarily on 
implementation at the program or district-level but also 
included reflections on implementation at individual schools. 


111.3.2.4. Implementation Support Data 

As mentioned in Section II, the evaluation team worked with 
the project staff to identify indicators with targeted levels 

of implementation for each of the Key Components shown 

in the logic model; these indicators served as measures of 
Fidelity of Implementation (FOI). Individuals responsible for 
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Table III-19. Qualitative Interviews Summary 


Poledvoe Year 2 Year 3 Year 4 
yp (2017-18) (2018-19 (2019-20) 
3 


Columbus State CC Staff (project leads and advisors) 


District Staff (coordinators and CCRE Cabinet members) Nl 7 1 
District Instructional Coaches 4 1 2 
Principals and Assistant Principals 9 7 9 
Counselors 9 8 8 
Teachers 17 19 17 
Students (in focus groups) 37 31 21 
Site-Based Advisors (I Know | Can and Communities in Schools) (0) 6 6 
Total 99 88 87 
providing the implementation supports entered these data After the final site visits were complete, research team 
into an online spreadsheet that was used for tracking project members used the collected information, coupled with 
implementation. These implementation data were analyzed information from the survey, project records, and other 
annually to assess FOI (results are included in Appendix A). interviews, to create a case study write-up for each school. 
These data were also used in the case studies to examine Each write-up, which averaged about 25 pages, included 
levels of participation in program supports. the school’s background information, a description of the 


implementation supports the school received, a description of 


111.3.2.5. Staff Survey Data how the school began the work of the project, a description of 


The staff survey was described in Section HL2, above. how the school implemented the Design Principles over time, 


In addition to looking at changes across the entire set of perceptions of what the impact of the project has been on the 
schools, we developed school-level survey reports that were school and a discussion of how the school staff were thinking 
incorporated into the case studies. about sustainability. The team then conducted a cross-case 
analysis that summarized patterns of implementation across 
11.3.3. Analyses the six schools. Results from this cross-case analysis are 
Following each site visit, the interviews were transcribed included in Section V. 

and entered into Atlas.ti. SERVE staff coded the transcripts 

using prespecified codes that were aligned to aspects of 

implementation. Each year the team jointly coded a set of 


transcripts to ensure common understanding for the codes. 
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Section IV: Impacts on Students 


The intent of CCRE was to increase the college and career 
readiness of students by implementing the Early College 
Design Principles. The study assessed the extent to which 
the project was successful by looking at the impact of CCRE 
on acore set of high school student outcomes: 1) students’ 
successful progression in a college preparatory course of 
study, 2) the number of students staying in school, and 3) 
student enrollment and success in college-level courses. This 
section presents results from the quasi-experimental impact 
study. Key findings presented in this section include: 


e There was a statistically significant positive impact of 
more than five percentage points on the percentage 
of students who were performing well on exams 
associated with successful progression in a college 
preparatory course of study. 


e There was no impact on the percentage of students 
dropping out. 


e There was a statistically significant positive impact 
on the percentage of students enrolled in college 
credit courses but not a significant impact on the total 
number of credits earned. 


Table IV-1: College Readiness Outcomes 


Years Assessed 


Covel exolanl-Me(-yilalinieya) 


As described in more depth in the methodology section, 

the impact study used a quasi-experimental design in 

which CCRE schools were matched to non-participating 
schools. Both sets of schools started with similar levels 

of the outcomes and with similar student demographic 
characteristics (see Table ITI.2 in Section ITI for the baseline 
equivalence). Each student sample also exhibited baseline 
equivalence on key characteristics. Section III also describes 
how the measures for each outcome were defined and 
created; here, we will include some of the key points, but 
readers are encouraged to refer to the methodology section 
for more detail. 


IV.1. IMPACT ON SUCCESSFUL COMPLETION IN A 
COLLEGE PREPARATORY COURSE OF STUDY 

This outcome explored the extent to which CCRE increased 
the number of students who are ready for college and career. 
We looked at this at two points in time, ninth grade and tenth 
grade. Table IV-1 summarizes how the two outcomes are 
defined by time point. 


It is important to note that we could not look at these 
outcomes in the 2019-20 school year because no end-of- 


F-Taal ol (-MPY-y a lali diel a] 


% of students earning at least five 


9th 2017-18; 2018-19 


graduation points on end-of-course 
exams with at least two in Algebra | or 
Math | and at least two in English | 


All 9th graders with at least .95 ADM? at 
a sample school or who dropped out of 
a sample school in the given year 


% of students earning at least 15 


10th 2018-19 


exams with at least four points in math, 
English, and science/social studies 


graduation points on end-of-course 


All 10th graders with at least .95 ADM at 
a sample school or who dropped out in 
the given year 


2 Average Daily Membership (ADM), which designates the proportion of the school year students were enrolled at each Ohio public school 
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course exams were administered in the spring of 2020 due to students or students not from underrepresented race/ethnic 

the COVID-19 pandemic. groups in either the baseline or the outcome year, which led 
to subgroup impact estimates with means below the overall 

IV.1.1. Impact on Ninth Grade College Readiness impact estimate. The number of schools included in 

The confirmatory analysis for Outcome 1 measured the the sample is included under the student sample size 

impact of students earning enough graduation points to be in Table IV-3. 


considered “on track” for high school graduation by the end of 
ninth grade. As shown in Table IV-2, there was a statistically We also tested impacts on students who did not earn any EOC 


significant, positive impact on ninth grade students’ on-track graduation points prior to attending high school (such as 
status for the pooled (or combined two-year) sample, as well by taking Algebra I in eighth grade). This was an important 
as in each individual year. The combined impact of Years 2 subgroup to test to eliminate a potential alternative 
and 3 of 5.8 percentage points represents 449 more students explanation for the results—that the impacts were driven 
on track in the treatment schools than in the comparison by treatment students coming to ninth grade more likely 
schools across the two school years. to have earned graduation points before high school than 
their comparison counterparts. The results in Table IV-4 
Subgroup Analyses. We ran subgroup analyses on the (page 40) show that this was not the case; the subsample of 
combined Year 2 and Year 3 samples to see if there students with no pre-high school graduation points saw a 6.1 
were differential impacts on students from different percentage point impact. In addition, the share of students 
demographic groups. We looked at five subgroups: 1) all who earned graduation points in eighth grade was very close 
students, 2) economically disadvantaged students, 3) not in the two groups of schools: 20.2% in the treatment group 
economically disadvantaged students, 4) students from and 19.6% in the comparison group. 
underrepresented racial/ethnic groups, and 5) students 
not from underrepresented racial/ethnic groups. Table IV-3 IV.1.2. Impact on Tenth Grade College Readiness 
(page 40) summarizes the outcomes by subgroup. We found The confirmatory analysis for Outcome 2 corresponds to the 
little variance in subgroup impacts, which ranged from 4.0 impact of students earning enough graduation points to be 
to 5.0 percentage points. As a reminder, some schools in the considered “on track” for high school graduation by the end 
sample did not have any non-economically disadvantaged of tenth grade. As shown in Table IV-5 (page 40), there was 


Table IV-2. Summary of Ninth Grade College and Career Readiness Impacts in Years 2 &3 
(Pooled, 2017-18 & 2018-19) 


Treat N Comp N Adjusted Unadjusted 
(in 16 (Cape Treatment | Comparison 
Yoda Lote) [9] Tol aLofo) [9] Mean Mean 


Tan) ex=\oa Effect Size 


Estimate | (Cox’s Index) 


9th Grade On Track (Pooled) 7,749 14,481 60.2% 54.4% +5.8 pp** 10 SD .003 
eee 3,919 7,328 60.8% 55.5% 5.3 pp* 09SD 021 
R : +5. : ; 
in Year 2 (2017-18) : , : is a 
9th Grade On Track 
3,830 7,153 59.3% 53.3% +6.0 pp** 11SD .003 


in Year 3 (2018-19) 


*Statistically significant at p < .05; **Statistically significant at p < .01. Main impact was statistically significant after adjusting for multiple comparisons. 
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Table IV-3. Summary of Ninth Grade On-Track Impacts on Combined Sample in Years 2 &3 
(2017-18 & 2018-19), by Subgroup 


Adjusted Unadjusted Effect 
Treatment | Comparison| Impact _ | Size (Cox’s 
18] ofe] co} 0] 0) Treat N Comp N Mean Mean Estimate Tate {=°4) 
7,749 14,481 
All Students 60.2% 54.4% +5.8 pp** 14 SD .003 
(16 sch) (32 sch) 
: : 5,074 9,474 
Economically Disadvantaged 46.7% 42.1% +4.6 pp* 1SD .022 
(16 sch) (32 sch) 
. ‘ 2,675 4,973 
Not Economically Disadvantaged 82.8% 77.8% +5.0 pp* 19 SD .010 
(12 sch) (18 sch) 
ra 3,684 7,166 
Underrepresented Race/Ethnicity 43.1% 38.5% +4.6 pp* 12 SD .022 
(16 sch) (32 sch) 
is 4,065 7,299 
Not Underrepresented Race/Ethnicity 74.0% 70.0% +4.0 pp 12 SD .060 
(16 sch) (29 sch) 


*Statistically significant at p < .05; **Statistically significant at p < .01. 


Table IV-4. Summary of Ninth Grade On-Track Impacts on Students with No Pre-High School 
Graduation Points - Combined Sample in Years 2 & 3 (2017-18 & 2018-19) 


Adjusted Unadjusted 
Treat N Comp N Treatment Comparison Tan) ex-\oad Effect Size 

10] oe] cole] 0) (16 sch) (32 sch) Mean Mean Estimate { (oop Sa Tate [=> 4] 
All Students 7,749 14,481 60.2% 54.4% +5.8 pp** 10 SD .003 

No Pre- 
High School 

. 6,186 11,649 50.9% 44.8% +6.1 pp** NSD .004 

Graduation 

Points 


*Statistically significant at p < .05; **Statistically significant at p < .01. 


Table IV-5. Summary of Tenth Grade On-Track Impacts in Year 3 (2018-19) 


Adjusted Unadjusted Effect 
Treatment Comparison Tray ex-Kea Size (Cox’s 
STU] ofe] oJ] 0) Treat N Comp N Mean Mean Estimate Tate {=>°4) 


10th Grade On Track in 3,422 6,582 


56.9% 50.4% +6.5 pp*** 16 SD <.001 
Year 3 (2018-19) (16 sch) (32 sch) 3 : os, 


*Statistically significant at p < .001. Main impact was statistically significant after adjusting for multiple comparisons. 
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Table IV-6. Summary of Tenth Grade On-Track Impacts in Year 3 (2018-19), by Subgroup 


Adjusted 
Treatment 
S10] ofe Tce} 0) Treat N Comp N Mean 
3,422 6,582 
All Students 56.9% 
(16 sch) (32 sch) 
Economically 2,111 4,056 
; 43.4% 
Disadvantaged (16 sch) (32 sch) 
Not Economically 1,311 2,526 
: 79.0% 
Disadvantaged (11 sch) (19 sch) 
Underrepresented 1,584 3,099 
. 39.3% 
Race/Ethnic (16 sch) (32 sch) 
Not Underrepresented 1,838 3,477 
i 71.7% 
Race/Ethnic (16 sch) (29 sch) 


Unadjusted Effect 

Covey nn) es-Jaxeyal Tony ey-Loa Size (Cox’s 
Mean Estimate Take {=>-4] p Value 
50.4% +6.5 pp** J16SD <.001 
37.2% +6.2 pp** J16SD <.001 
71.6% +7.4 pp* 245D .033 
33.3% +6.0 pp* J16SD 005 
65.6% +6.1 pp* J17SD 004 


*Statistically significant at p < .05; **Statistically significant at p < .01,; ***Statistically significant at p < .001. 


a statistically significant positive impact of 6.5 percentage 
points on tenth grade students’ on-track status in Year 3. 

The 6.5 percentage point impact represents an estimated 
additional 222 students on track in the treatment schools as 
compared to the comparison schools for this cohort. It should 
be noted that many of the students in the tenth grade sample 
in 2018-19 were also part of the ninth grade sample in 2017- 
18, which saw a similar magnitude impact for ninth grade on- 
track status. This suggests that students who were on track in 
ninth grade remained on track in tenth grade as well. 


Subgroup Analyses. Table IV-6 summarizes the outcomes 

by subgroup. The impacts by subgroup were consistently 
positive, ranging from 5.5 to 6.7 percentage points. The table 
also shows that there were significant gaps in the rates 
between different subgroups. For example, economically 
disadvantaged students were much less likely to be on track 
than non-economically disadvantaged students. 


IV.2. IMPACT ON DROPPING OUT OF SCHOOL 
CCRE was intended to improve students’ engagement in 
school and thus, potentially, keep more students in school. 
‘We measured this by looking at the percentage of students 
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who dropped out of school in the 2018-19 school year; data 
for the 19-20 school year were not available as of the writing 
of this report. The confirmatory outcome utilized the pooled 
sample (the two years combined); we also report the findings 
for each year separately. As a reminder, this sample included 
students who were missing baseline achievement scores with 
baseline equivalence shown at the school-level on a previous 
cohort of students (see section IT1.1.2.3 for a more in-depth 
discussion of the outcome and sample definition and see 
Table B-3 for representativeness of this sample). 


The confirmatory analysis for Outcome 3 corresponded to 
the impact of students dropping out of school in 2018-19. 
Table IV-7 (page 42) summarizes the findings. Although 

the estimated rate of dropping out was lower in the CCRE 
schools by 0.7 percentage points, there was not a statistically 
significant program impact on dropout rate. Note that in the 
tables about dropout rates that a negative impact on the rate 
is better, signaling a lower dropout rate. 


Subgroup Analyses. We ran a sensitivity analysis of the 
dropout outcome for all students with non-missing 
data (baseline equivalence is shown for the students in 
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Table IV-7. Summary of Dropout Impacts in Year 3 (2018-19) 


Unadjusted 
Treat N Comp N Adjusted Treat-| Comparison Impact Effect Size 
(16 sch) (32 sch) ment Mean Mean Estimate { (oC MT Te {=°4) 
Dropping 
Out of School 14,875 27,845 3.1% 3.8% -0.7 pp -12 SD 470 
(2018-19) 


No differences were statistically significant. 


Table IV-8. Summary of Dropout Impacts on Subgroups in Year 3 (2018-19), by Subgroup 


Adjusted Unadjusted 


Treatment | Comparison Impact Effect Size 
Subgroup Treat N Comp N Mean Mean Estimate | (Cox’s Index) 

All Students with No 13,382 25,593 2.9% 3.5% 06 -2SD 480 
Missing Data’ (16 sch) (32 sch) i = ore ; . 
Economically 8,205 16,054 41% 4.7% 06 -08 SD 495 

Disadvantaged (16 sch) (32 sch) ie si mee : , 
Not Economically 5,177 9,539 15% 14% +01 03SD 873 
Disadvantaged (12 sch) (20 sch) i aia oe . . 
Underrepresented Race/ 6,163 12,353 3.49 44% 10 -17SD 286 
Ethnic (16 sch) (32 sch) Cs — ee 
Not Underrepresented 7,219 13,240 23% > 6% 03 -03SD 738 
Race/Ethnic (16 sch) (29 sch) ae ee a . , 
No differences were statistically significant. 
this analytic sample in Table III-10). We also conducted IV.3. IMPACT ON COLLEGE COURSE ENROLLMENT 
subgroup analyses using students with non-missing A key goal of CCRE was increasing the number of students 
data to see if there were differential impacts on students enrolling in college courses. This outcome looked at the 
representing various demographic characteristics. We percentage of students in grades 10-12" taking any college- 
looked at five subgroups: a) all students with non-missing level course, including both dual-enrollment courses and AP 
data, b) economically disadvantaged, c) not economically courses. We examined this for both the 2018-19 and 2019-20 
disadvantaged, d) underrepresented race/ethnic groups, school years. The methodology section describes how the 
and e) not underrepresented race/ethnic groups. Unlike the outcome and sample were defined in more depth. 
confirmatory analysis, we ran each of these analyses only for 
students with no missing data. Table IV-8 summarizes the The main confirmatory analysis for college course 
outcomes by subgroup. Like the confirmatory analysis, none enrollments corresponded to the impact of students taking 
of the subgroup analyses were statistically significant. either CCP or AP courses. As summarized in Table IV-9, 


10 This sample was different than the sample for the confirmatory analyses in Table IV-7, which is why the impact estimates are slightly different. 
As a reminder, the project looked only at grades 10-12 because of issues with a new 9th grade school forming in one of the districts in the middle of the study. More information can be found in Section III.1.2.4. 
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Table IV-9. Summary of College Course Enrollment Impacts in Years 3 & 4 (Pooled Sample and by Year) 


Adjusted 
Treatment 
Mean 


Bg-y-1 a) | Comp N 


i CY-Tod a) 


(16 sch) 


Effect Size 
(Cox’s 
Tate {=>°4} 


Unadjusted 
Comparison 
Mean 


Impact 
Estimate 


Took Either CCP 


20,929 41,628 
or AP (pooled) 
Took Either CCP 

10,192 20,524 
or AP (2018-19) 
Took Either CCP 

10,737 21,104 


or AP (2019-20) 


29.9% 


29.4% 


30.3% 


22.7% 7.2pp* .23SD .003 
22.2% 7.2pp* .23SD 004 
23.2% 7.1pp** 22SD 005 


*Statistically significant at p < .01. Main impact was statistically significant after adjusting for multiple comparisons. 


we found a 7.2 percentage point impact on college course 
enrollment in CCRE schools, with an adjusted 29.9% of the 
treatment group and 22.7% of the comparison group for 
students in Grades 10-12 taking college courses in 2018-19. 
This outcome was significant at the p < .01 level with an effect 
size of .23 SD. The individual analyses for Years 3 and 

4 yielded nearly identical impact estimates; the increase seen 
from Year 3 to Year 4 in comparison school college course- 
taking of approximately one percentage point paralleled a 
similar adjusted increase in the CCRE schools suggesting 
that any additional expansion occurring between Years 3 

and 4 was likely due to the statewide policy context. 


‘We also looked at each of the college-level course options, 
CCP and AP, individually for the pooled sample (which 
combined the two years of data). Summarized in Table IV-10, 
we found a 7.3 percentage point impact on CCP coursetaking 
in CCRE schools, a .32 SD effect size significant at the p < 
.001 level. There was not a statistically significant impact 

on AP course enrollment, which is not surprising given 

that AP courses were not an explicit focus of the CCRE 
program. However, a non-negative (and positive, although 
not statistically significant) finding is important because 
increases in students taking college courses through CCP 
were not at the expense of AP course enrollments. 


Table IV-10. Summary of College Course Enrollment Impacts, by Type of Course (Pooled Sample) 


Adjusted 
Treatment 


Mean 


Effect Size 
[ (eo 3 
Tae (=>.4] 


Unadjusted 
Comparison 
Ta) 


Impact 
Estimate 


Treat N Comp N 

(16 sch) (32 sch) 
Took Either CCP or AP 

20,929 41,628 
(pooled) 
Took CCP (pooled) 20,929 41,628 
Took AP (pooled) 20,929 41,628 


*Statistically significant at p < .01.; ***Statistically significant at p < .001. 


29.9% 


20.7% 


14.5% 


22.7% +7.2pp* .23SD .003 
13.4% +7.3pp** 32SD <.001 
13.5% +1.0pp 05 SD 653 


” The subgroup results produce a somewhat surprising result in that two mutually exclusive groups (e.g., economically disadvantaged and non-economically disadvantaged) can yield impact estimates that are both larger than the total. This difference is not 
an error but is due to the statistical adjustments to the treatment group proportion that take place in each impact model as well as not all schools having students who are part of every subgroup. 
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Subgroup Analysis. The sample for the college course 
enrollment outcome contained students from multiple grade 
levels and subgroups. For each outcome, we ran 15 subgroup 
analyses representing three different grade level ranges: 

1) Grades 10-12, 2) Grades 11-12, and 3) Grade 10, and five 
subgroups: 1) all students, 2) economically disadvantaged 
students, 3) not economically disadvantaged students, 4) 
students from underrepresented racial/ethnic groups, and 

5) students not from underrepresented racial/ethnic groups. 
Table IV-11 presents the results for our confirmatory outcome 
of students taking either CCP or AP courses in grades 10-12 
by demographic subgroups. As the table shows, the subgroup 
impact estimates ranged from 6.8 to 9.6 percentage points. 
‘We also looked at impacts for each grade span by the different 
demographic sub-groups. The number of analyses involved 
makes the tables large; the results can be found as Tables 
D.1-D.3 in Appendix D and the findings are summarized 
below. 


Students Taking Either CCP or AP Courses. Table D.1 displays 
the results for students taking either CCP or AP courses in 
2018-19. The impacts for all subgroups were statistically 
significant. For Grades 10-12, the percentage point 
differences were relatively consistent across all subgroups, 
ranging from 6.8 to 9.6 percentage points respectively.” 

The comparison of effect sizes across subgroups for all 

grade level designations also shows larger impacts for 
economically disadvantaged students and students from 
underrepresented racial/ethnic groups compared to non- 
economically disadvantaged students and students not from 
underrepresented racial/ethnic groups. We interpreted these 
effect sizes as showing that although the CCRE treatment 
schools had positive impacts in college course enrollment 

for all students, the impact on targeted groups was larger 

in magnitude than in non-targeted groups. For students in 
Grades 10-12 the effect sizes for economically disadvantaged 


Table IV-11: Students Taking Either CCP or AP (Pooled Sample), by Subgroup 


Adj. Treat 


Mean 


Effect Size 
{ (oC y eM Tate {=°4) 


Unadj. 
Comp Mean 


Impact 


Estimate p Value 


Population Treat N Comp N 
20,929 41,628 
All Students (Grades 10-12) 
(16 sch) (32 sch) 
13,216 26,638 
All Students (Grades 11-12) 
(16 sch) (32 sch) 
7,713 14,990 
All Students (Grade 10) 
(16 sch) (32 sch) 
Economically 
; 12,758 24,685 
Disadvantaged 
(16 sch) (32 sch) 
(Grades 10-12) 
Not Economically 
. 8,171 15,886 
Disadvantaged (Grades 
(12 sch) (21 sch) 
10-12) 
Underrepresented Race/ 9,928 20,126 
Ethnic (Grades 10-12) (16 sch) (32 sch) 
Not Underrepresented 11,001 21,458 
Race/Ethnic (Grades 10-12) (16 sch) (29 sch) 


*Statistically significant at p < .05; **Statistically significant at p < .01. 


29.9% 


36.3% 


18.8% 


22.8% 


43.4% 


21.2% 


39.2% 


22.7% +7.2pp* 0.225 0.003 
29.1% +7.3pp* 0.201 0.0n1 

11.4% +7.4pp** 0.356 0.003 
15.1% +7.7pp** 0.307 0.002 
33.7% +9,.6pp* 0.248 0.008 
14.4% +6.8pp** 0.286 0.005 
30.5% +8.7pp* 0.232 0.004 
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students (.381 SD) and students from underrepresented 
racial/ethnic groups (.29 SD) were slightly larger than those 
for non-economically disadvantaged students (.25 SD) and 
students not from underrepresented racial/ethnic groups 
(.23 SD) students. Similarly, effect sizes for tenth grade 
students were in general larger than those for students in 
grades 10-12. 


Students Taking Only CCP Courses. Table D.2 displays the 
results for college course enrollment for students who only 
took College Credit Plus courses. There were statistically 
significant impacts for all subgroups except for non- 
economically disadvantaged students. Note that the data 
included CCP courses taken at any postsecondary institution; 
although most treatment students took their CCP courses at 
Columbus State, the results capture courses taken at other 
institutions as well. Similar to the results for the overall 
college course enrollment measure, we found larger effect size 
impacts for the target groups of economically disadvantaged 
students (.56 SD) and students from underrepresented 
racial/ethnic groups (.49 SD) than for the non-targeted 
groups (.16 SD and .28 SD, respectively). 


Students Taking Only AP Courses. The final analysis 

explored subgroup impacts on AP course enrollment, 
summarized in Table D.3. Overall, the AP results did not 
show much difference between the treatment and comparison 
schools. However, all impact estimates for both Grades 10-12 
and Grades 11-12 were positive, meaning that the adjusted 
proportion of students taking AP courses in the CCRE 
treatment schools was higher than those in the comparison 
schools. We found statistically significant impacts on 

AP coursetaking for non-economically disadvantaged 
students. The impact of 7.1 percentage points on AP course 
enrollment for non-economically disadvantaged students in 
grades 10-12 was significant at p <. 05. This is an interesting 
finding since there was a null impact on CCP coursetaking for 
this subgroup. 
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IV.4. IMPACT ON COLLEGE CREDITS EARNED 
While the previous outcome focused on changes in student 
access, this outcome examined the impact on students 
earning college credit. Students could receive college credit 
either by successfully completing a College Credit Plus 
course or by scoring a 3 or higher on an Advanced Placement 
exam. The confirmatory analysis focused on the impact of 
the average combined number of college credits earned from 
CCP and AP for students graduating from high school in 
2018-2019 and in 2019-20. To better understand patterns 

of students earning college credits we also ran follow-up 
analyses to look at: 1) the average number of dual credits 
earned, 2) the average number of AP credit equivalents 
earned, 3) whether a student graduated with any dual credit, 
4) whether a student graduated with three or more dual 
credits, and 5) whether a student graduated with any AP 
credit equivalents. 


As described in more depth in the methodology section, the 
Ohio Department of Education only collects data on course 
credits earned at the point of a student’s graduation from high 
school. As a result, the sample was restricted to high school 
graduates. This means that the sample for college course 
enrollment and the sample used for college credits earned was 
not the same, making comparisons between the two outcomes 
challenging, although we did attempt some comparisons, as 
described below. 


The number of credits earned through dual enrollment was 
reported in the high school graduation file. To determine the 
number of credits earned through an AP exam, we used a 
crosswalk of AP college credit equivalents used to determine 
the number of college credits students would earn for different 
AP scores at Ohio University. 


The confirmatory analysis looked at the total average 
number of college credits earned (through dual credit and 
through AP equivalents) by graduating students in sample 
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schools in 2018-19. Shown in Table IV-12, on average, 
graduates in CCRE schools earned fewer total college 
credits and dual credits than graduates in the comparison 
schools. Graduates of CCRE schools earned slightly more AP 
credit equivalents. None of these differences, however, were 
statistically significant. 


‘When we looked at the number of college credits earned by 
type of credit, we saw no statistically significant differences 
(Table IV-18, page 47). 


We also conducted some additional exploratory analyses, 
measuring additional outcomes related to the number 

of college credits earned by the graduates in both years. 
These outcomes were students 1) earning any dual credit, 

2) earning three or more dual credits, 3) taking an AP exam, 
and 4) earning any college credit equivalents by earning a 

3 or higher on an AP exam. As shown in Table IV-14 (page 47), 
there was a statistically significant impact on the percentage 
of students earning any college credits or any dual credit but 


no impact on the percentage earning 3 or more credits or 
any AP credits. 


Subgroup Analyses. When we looked at the impact for 

the confirmatory outcome—number of credits earned—by 
subgroup, we saw a similar pattern to the overall findings 
with no statistically significant differences, Table IV-15 
(page 48). 


IV.5. COMPARING COLLEGE COURSETAKING 
AND CREDITS EARNED 

The finding that CCRE expanded access to college courses 
for more students but did not increase the overall number of 
credits earned deserved further exploration. As a result, we 
conducted additional analysis using the 2018-19 graduates 
to better understand the connection between students 
taking college-level coursework and students earning credit 
in college-level coursework, represented by Outcomes 

4 and 5 respectively. The data limited us to only twelfth 
grade students who graduated high school, but this extra 


Table IV-12. Summary of College Credits Earned, Combined CCP and AP 


Treat N (eCoynal omy | 


Average College Credits Earned 6,341 13,039 
(Combined CCP and AP, pooled) (16 sch) (32 sch) 
Average College Credits Earned 3,060 6,424 
(Combined CCP and AP, 2018-19) (16 sch) (32 sch) 
Average College Credits Earned 3,281 6,615 
(Combined CCP and AP, 2019-20) (16 sch) (32 sch) 


No differences were statistically significant. 


Adjusted 


Treatment | Comparison 


Unadjusted Tan) ex-\oa 
Estimate 


((={=\ #2) 


Effect Size 


Mean Mean (Hedges’ g) p Value 


4.68 5.45 
Credits Credits -0.77 credits -.07 SD 3569 
(SD=9.68) (SD=11.55) 
4.90 551 
Credits Credits -0.61 credits 3-.06 SD 533 
(SD=9.55) (SD=11.16) 
5.40 
4.55 ; 
; Credits ; 
Credits -0.85 credits -.09 SD .287 
(SD=11.92) 
(SD=9.81) 
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Table IV-13. Summary of College Credits Earned (Pooled Sample), by Type of Credit 


Adjusted | Unadjusted Impact 


Treatment |Comparison| Estimate Effect Size 


Treat N Comp N Mean Mean [(={=¢-)) (Hedges’ g) 
. 4.68 5.45 
Average College Credits Earned 6,341 13,039 . . ; 
: Credits Credits -0.77 credits -.07 SD 569 
(Combined CCP and AP, pooled) (16 sch) (32 sch) 
(SD=9.68) (SD=11.55) 
; 3.62 4.4] 
Average CCP Credits earned 6,341 13,039 : . -0.79 
credits credits ; -.08 SD 379 
(pooled) (16 sch) (32 sch) credits 
(SD=8.39) (SD=10.20) 
; 1.06 1.04 
Average AP Credits Earned 6,341 13,039 ‘ . +0.02 
credits credits : .0O0 SD 935 
(pooled) (16 sch) (32 sch) credits 
(SD=3.67) (SD=4.51) 


No differences were statistically significant. 


Table IV-14. Summary of Additional Impacts Related to College Credit (Pooled Sample) 


Adjusted | Unadjusted Taal ex=\oas Effect 
Treatment |Comparison| Estimate Size (Cox’s 
Treat N Comp N Mean Mean (= f=) #2) Tae (=>.4] 
HS Graduates Earning Any 6,341 13,039 
. 37.3% 32.0% +5.3pp* 14 SD 041 
College Credit (CCP or AP) (16 sch) (32 sch) 
HS Graduates Earning 6,341 13,039 
. 33.9% 27.9% +6.0pp* 17 SD .046 
Any Dual Credit (16 sch) (32 sch) 
HS Graduates Earning 6,341 13,039 
: 29.5% 274% +2.1pp .06 SD 510 
3+ Dual Credits (16 sch) (32 sch) 
HS Graduates Earning Equivalent 6,341 13,039 
; 10.8% 9.8% +1.0pp .06 SD 562 
College Credit from an AP Exam (16 sch) (32 sch) 


*Statistically significant at p < .05 
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Table IV-15. Average Total Credits Earned (Dual Credit and AP) for Graduates (Pooled Sample), by Subgroup 


Adjusted | Unadjusted 


Treatment | Comparison Impact Effect Size 
Subgroup Treat N Comp N Mean Mean Estimate | (Hedges’ g) 
: 5.45 
6,341 13,039 4.67 credits ; . 
All Students (pooled) credits -0.77 credits -.07 SD 569 
(16 sch) (32 sch) (SD=9.68) 
(SD=11.55) 
: : . 2.60 
Economically Disadvantaged 3,560 7,410 2.47 credits ; . 
credits -0.13 credits -.01SD 841 
(pooled) (16 sch) (32 sch) (SD=7.55) 
(SD=7.79) 
. . : 9.55 
Not Economically Disadvantaged 2,781 5,307 9.00 credits ; , 
credits -0.56 credits -.05 SD .734 
(pooled) (12 sch) (19 sch) (SD=11.51) 
(SD=14.31) 
: . 2.62 
Underrepresented Race/Ethnic 2,915 6,239 2.08 credits ; -0.55 
credits . -.06 SD 529 
(pooled) (16 sch) (32 sch) (SD=7.74) credits 
(SD=8.11) 
. 8.07 
Not Underrepresented Race/ 3,426 6,785 7.82 credits : 0.24 
; credits ; -.02 SD .823 
Ethnic (pooled) (16 sch) (29 sch) (SD=10.92) credits 
(SD=13.48) 


No differences were statistically significant. 


analysis allowed us to better understand what is occurring. e Enrolled in a CCP Course, but no Dual Credit: Students 


To explore this outcome, we combined the data sets on 


college coursetaking and credits earned for the students in 


the pooled 2018-19 and 2019-20 graduation sample. This 
required us to use the course enrollment files for 2016-17 
to 2019-20 to identify graduates who had enrolled in one 


were assigned a 1 if they had enrolled in a CCP course 
and the sum of dual credits earned in the graduation 
core file was 0, and 0 otherwise. 


Earned Some Dual Credit, Fewer than 3 Credits: 
Students were assigned a 1 if the sum of dual credits 


or more CCP courses during the study period. These data earned in the graduation core file was greater than 0 


were combined with the total dual credits earned from the jah leas thaw 4 Badi0 otherwise: 


graduation core file. These combined data allowed us to 
define four dichotomous indicators—one related to Note that if a student had enrolled in a CCP course, they were 


enrollment and three related to the total dual credits assigned a score of 1 for one of the mutually exclusive dual 


earned—described below. credit categories and a score of 0 for the others. Students who 


never enrolled in a CCP course were assigned scores of 0 for 
e HS Graduates Enrolling in a CCP Course: Students 


were assigned a 1 if they had at least one CCP course 
enrollment record in 2016-17, 2017-18, or 2018-19, and 


all indicators. Thus, the percentage of students enrolling in a 
CCP course equals the sum of the percentage of students in 
each of the three credit-earning categories. 


0 otherwise. 

e HS Graduates Earning 3+ Dual Credits: Students were ‘We used these variables as outcomes in models using the 
assigned a 1 if the sum of dual credits earned in the same covariates as the other models (including baseline 
graduation core file was greater than or equal to 3, and values for these covariates from 2015-16). These analyses 
0 otherwise. showed that, in general, CCRE schools improved student 
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access to college-level coursework, including both CCP and 
AP courses, relative to comparison high schools. However, 
improved access did not lead to significant differences in the 
number of credits accrued or in the percentage of students 
earning three or more dual credits or scoring a 3 or higher on 
an AP exam to earn college credit equivalents. 


Figure IV-1 (page 53) details additional outcomes for the 
19,380 graduates (pooled for Years 3 and 4) who formed the 
analytic sample for Outcome 5. The figure illustrates the 
gaps in access versus credit earning outcomes. If we start 

by looking at the length of the two bars in the figure, we see 
that the treatment group had more high school graduates 
enrolling in dual-credit courses—an impact of 11.3 percentage 
points. Despite this positive impact on enrollment, we did 
not find a significant difference in the percentage of students 
earning three or more dual credits between the treatment 
and comparison schools. The different colored sections of the 
bars further explain what might be happening. 


Figure IV-1. Enrollment in CCP Courses and Earning 
Credits for 2018-19 and 2019-20 Graduates 


Treatment 
(Adjusted) 


0.5% 


ee fae i 


Wi Earned 3+ Dual Credits ™ Earned Some Dual Credit, 


Fewer than 3 Credits 


{> Enrolled in a CCP Course, 
No Dual Credit 
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First, a significantly higher proportion of students in CCP 
schools enrolled in a CCP course but did not earn any dual 
credit; this is shown as the orange section of the bar graph 
as a 4.6 percentage point difference between treatment and 
comparison schools (6.9% vs. 2.3%). The data did not have 
details about why students did not receive credit, so we 

did not know if a greater percentage of treatment school 
students were failing courses, or were withdrawing from 
courses, or if another phenomenon was occurring. Second, a 
significantly higher proportion of students in CCRE schools 
earned greater than 0 but fewer than 3 dual credits—also a 
4.6 percentage point difference between the groups (5.1% vs. 
0.5%). This result was likely driven by greater enrollments 
in only one-credit or two-credit courses, such as COLS 1101 
or BGMT 1008 at Columbus State, or similar courses at 
other institutions. Again, the data did not contain sufficient 
detail to discern what exactly was occurring—they did not 
include individual course names—but it appears that at the 
comparison schools, similar introductory courses were not 
being taken in large numbers without students also taking 
additional dual-credit courses. 
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Section V: Impacts on schools 


The CCRE Key Components (described in Section IT) 
supported implementation of the Early College Design 
Principles in the CCRE schools. The four Design Principles 
outlined in the grant proposal and logic model—a Career 

and College-Ready Academic Program, a College Headstart, 
Wraparound Student Supports, and School-Level 
Organizational Practices—describe characteristics of schools 
that are focused on career and college readiness. 


This section utilizes data from surveys, interviews, and 
observations to present findings related to school-level 
implementation of the four Early College Design Principles. 
Each section describes the Design Principle and how it was 
implemented over the life of the project. 


V.1. CAREER AND COLLEGE-READY 

ACADEMIC PROGRAM 

CCRE worked toward a bold goal of having 90% of high 
school students earn some college credit. The Career 

and College-Ready Academic Program Design Principle 

was the one most directly aligned to this goal. As part of 
this Design Principle, schools were expected to create an 
academic program of study that gave almost all students the 
opportunity to be prepared for college and to attain college 
credit while still in high school. In addition, schools were 
expected to create aligned sequences of courses or pathways 
that led to 12 or more college credits and that incorporated 
opportunities for work-based learning. This Design Principle 
also focused on classroom practices and instructional 
strategies that enhanced rigor. 


V.1.1. Coursework and Pathways 

A critical part of the Career and College-Ready Academic 
Program was to expand access to college-level courses. 

As the results presented in Section IV show, the schools 
were successful in expanding access to college courses. 
This section describes in more depth what that looked like 
at the school level. 


In the CCRE high schools, college-level courses were 
offered in one of four formats. First, students could take 
college courses on the Columbus State campus. Second, 
students could be taught on the high school campus by an 
adjunct faculty member, usually a high school teacher who 
met the qualifications set by Columbus State. Third, the 
college course could be a offered as a facilitated course. In 
the facilitated course setting, students were enrolled in a 
regular high school course (e.g., Chemistry) and would also 
participate in an online college course or receive additional 
instruction from a college instructor with the high school 
teacher acting as a facilitator. The fourth option was for 
students to take AP courses; to receive college credit, they 
also had to pass the exam associated with the course. 
These four formats allowed schools to provide college- 
coursetaking options in a way that best aligned with their 
needs and capacity. 


By the last full year of the project (2019-20), 37% of 
graduates in CCRE schools had earned at least some college 
credits (either CCP or AP). Data from Columbus State 
show that the number of dual-enrollment courses taken 

by students in CCRE schools more than doubled from the 
baseline year, increasing from 2,149 in 2015-16 to 5,218 in 
Year 4. This expansion did not result in lower pass rates, 
which remained fairly steady, even increasing slightly over 
time. Figure V-1 (page 51) shows the trends in Columbus 
State coursetaking since before the project through Year 4; 
it includes both the number of courses taken as well as the 
pass rates for those courses. 


A small proportion of this increase in coursetaking was due 
to increases in students taking two college readiness courses 
offered by many of the participating schools, COLS 1101 and 
BMGT 1008. The former was a one-credit college readiness 
course that was adapted from a course taught at Columbus 
State. The latter was a course that focused on 21st century 
workplace skills. At the baseline year, there were 112 of those 
courses taken, which increased to 667 in Year 4. 
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Figure V-1: Increases in Columbus State 
Dual-enrollment courses, 2015-2020 


{) Total Taken 
HB Total Passed 


5,213 


5,103 


2015-16 
(Baseline) 


2016-17 
(Year 1) 


2017-18 
(Year 2) 


2018-19 
(Year 3) 


2019-20 
(Year 4) 


More information about the strategies Columbus State and 
the schools used to expand access to college coursetaking 
can be found in the accompanying policy brief, Addressing 
Inequity: Expanding Access to College-Level Courses in High 
School. 


The program’s expectation was not just that schools should 
get students to take college courses but that these courses 
would be structured as pathways that would lead students 
to acollege major or a technical credential. By the end of the 
project, each school had submitted a design for at least one 


pathway, but they were in varying degrees of implementation. 


These pathways were created to address district and school 
needs. Some districts created pathways in line with the 
original CCRE focus on developing a sequence of high 

school and college courses that supported matriculation at 
Columbus State or another higher education institution. 
Other districts focused on building pathways with immediate 
CTE credentials. One district also conceptualized post- 

high school pathways in a broad sense, thinking about 

three different post-high school activities: postsecondary 
enrollment, employment, and enlistment in the military. 
Creating pathways was a heavy lift for many of the schools as 
they faced challenges in identifying areas of study that had 
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both sufficient student interest and courses that could be 
taught by current faculty, and that could transition students 
to college courses or a career. 


Many school staff noted that the expanded access to college 
courses was changing the culture of their schools and thereby 
increasing expectations for students. For example, when 


asked about the impact of the project, one teacher said, 


A teacher in a different school agreed, noting, 


[I see] more students taking the college 
courses, definitely. | mean, that's the 
biggest thing, and | think then that 
changes the mindset of teachers because 
... it's very rewarding and encouraging 
when you see your young people in your 
school doing well in these college courses, 
when you see them graduating with 
college credit and going on to college. 

I think that is something that was not 
happening when | first came to [this 
school]. And | think it gives credence to 
what we're saying when we're saying, 
"Hey, we have to prepare them for 
college,”... Because years ago, [this high 
school] had a low graduation rate, and the 
students weren't going to college. And 
So, it's easy to say, "Oh well these kids 
aren't going to do that." But when you 
see students doing it, | think it changes 
the mindset and it makes people rise up a 
little bit more. 


— Staff member in district 


“Tt's shifted some of that mindset too, that focusing on, ‘What 
can we do to get students college credit?’ [It’s] to the point 
now where we're offering AP courses at the freshman level.” 
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V.2.2. Work-Based Learning 

To increase the relevance of students’ high school 
experiences and make a better connection to careers, CCRE 
schools were expected to provide work-based learning 
experiences in the school associated with the pathways. 
Columbus State staff conceptualized work-based learning as 
a continuum of activities that ranged from career awareness 
activities, to high school and college coursework aligned with 
career pathways, to meaningful work experiences, such as 
career assessments, job shadowing, and internships. 


Table V.1. Work-Based Learning—Year 1 to Year 4 


Work-based Year 4 Mean 


(aba I-)) 


Year 1 Mean 
(n=15) 


R-¥-T dal late 


Overall Mean 2.38 2.56 


1- Not offered; 2- Fewer than 25%; 
3- 25% to 50%; 4- 51% to 75%; 
5- Greater than 75% 


Scale 


No differences were statistically significant. 


In general, schools showed less change in their work-based 
learning activities than in other areas. The survey asked 
school administrators and counselors to indicate the 
percentage of students who participate in various work-based 
learning activities before they graduate. The results showed 
an increase in the implementation of work-based learning 
activities but with a non-significant difference between Year 1 
and Year 4; the lack of significance may be driven by the very 
small sample size given that analysis of this scale was at the 
school level. Table V.1 shows the overall scale results. 


The reported lack of a large positive impact on work-based 
learning was supported by the interviews, which suggested 
that the implementation of work-based learning activities 


was still in the beginning phases, even in Year 4 of the project. 


Interviewees reported activities related to career awareness, 
such as using the software package Naviance, hosting 
career days, or inviting local employers in as guest speakers. 
Teachers in one district reported integrating soft skills and 
work-based content into their instructional practices. A 
student in that district described how her business pathway 
had given her exposure to real-world experiences: 


“So, my junior year, we learned how to 
start a business up and maintain our 
business. And our business for the school 
is making t-shirts and making designs 
for everybody. And so, now our senior 
year, we're doing the Incubator, which is 
like, we make basically an invention. My 
product is a saving app to help kids save 
their money. So, if it goes big, after high 
school, we can have that as our job and 
sell that product.” 


— Student 


Establishing more resource-intensive activities, such as 
internships or job-shadowing, was more challenging for 
schools. Most districts reported that they were in the early 
stages of trying to make connections with employers and 
setting up meaningful work-based learning opportunities. 
Two schools were trying to get internship experiences up and 
running, but they were not yet fully operational. A counselor 
in one district believed that the grant had not helped with 


work-based learning: 


“I feel like that's a big deficit, and this 
grant has done great for the 5% of kids 
who are the top of the class and are ready 
to do their CCP and are college-bound, 
and they just got it. But we have 50% of 
the class who are going to be amazing 
workers, but we're not cultivating that; 
when you said, “work-based experience,” 
our kids would love to do that ... | think 
that a lot of our kids could benefit from a 
dual enrollment program, that really does 
that apprenticeship. So no, I haven't seen 
a change.” 


— Counselor 
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On the other hand, a teacher in another district 
acknowledged the grant had started out with the college- 
going focus, but had shifted to also include more about work- 
based learning: 


“I think what we noticed was... when 

we first started doing the college 
readiness initiative, [we were] forgetting 
the fact that there were students that just 
realistically we're not going to be on that 
[college] path, and we didn't want them to 
feel isolated or left out. And so, | think that 
the way we're doing it now is a lot more 
inclusive.” 


— Teacher 


V.1.3. Instructional Practices 

The CCRE program focused on high school classroom 
instruction as a major contributor to student college and 
career readiness. As such, a shared vision for instruction 
was seen as an important part of a school-wide Career and 
College-Ready Academic Program. According to the survey 
results, 52% of Year 4 respondents indicated that there was 
a common vision driving major instructional decisions for 
all staff; this was a statistically significant increase over 
the Year 1 level of 36%. The survey results were supported 
by interviews. Participants in three of the site visit schools 
commented that they were using a common framework to 
support instructional improvement. As an administrator in 
one of the schools said, “I would say common language was 
huge—everyone having the same common language when 
you're talking about instruction.” 


The original intent was that all schools implement the 
Common Instruction Framework (or CIF), a set of six 
instructional practices articulated by JFF. Table V-2 (page 
54) shows the CIF strategies and how they were measured 
through the school implementation survey. As seen in the 
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table, the survey measured no significant changes in school- 
wide presence of the CIF strategies between Year 1 and Year 
4, 


Interviews suggested that the lack of significant changes 
across the board was likely because the emphasis on 
instructional practices differed by district. One district had 
an extremely strong emphasis on instructional change, 

with the CIF serving as one of the organizing structures 

for district-wide school improvement. In this district, all 
staff received formal professional development in the CIF 
strategies, and department chairs were expected to ensure 
that teachers in their departments were implementing the 
CIF strategies as well as model the instructional practices in 
their own meetings. According to an administrator, the CIF 
strategies became embedded in the culture of the school and 
the district by the end of the project: 


“We're now into the fourth year that | 
think it is part of the culture. It's part of 
our framework for learning. All staff are 
aware, and ... | don't think there's the 
pushback on it at all. It is just an accepted 
Way ... particularly for the instructional 
piece, that is how we teach. It's wrapped 
up into every building's professional 
development plans, staff retreats, our new 
teacher orientation programs, and even 
like, your one [to] three teachers whom 
are relatively new to the district undergo 
training.” 


—Administrator 


The staff in this district believed that implementation of the 
CIF had a positive impact on students. For example, one 
teacher said, “[Student learning] has definitely increased 
because they're more responsible for their own learning; they 
have to figure it out.” 
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Table V-2. Use of CIF Practices—Year 1 to Year 4 


Year 1 Mean 


(ak 4-10) 


Year 4 Mean 
{ (a 4-1-9) } 


FTanl o)(-Mele(-sad(e) a) 
This year how frequently have you... 


Classroom Talk 


: 3.78 
(5 items) 
Collaborative Group Work 

: 3.60 
(3 items) 
Literacy G 

y Groups 328 
(2 items) 

tioni 

ques ioning 37 
(3 items) 
Scaffolding 

: 3.95 
(4 items) 
Writing to Learn 

. 3.60 
(3 items) 
Overall Mean 3.70 


Asked students to engage in in-depth 


3.83 discussions about what they have read or 
learned? 
357 Defined clear roles and expectations for 
: students working in groups? 
329 Had students read content-based texts in 
? groups? 
374, Used student-developed questions to guide 
. discussions? 
4.01 Made connections between material covered 
, previously and new content? 
Used low-stakes writing (e.g., journaling, 
3.69 free writing, open-response exit tickets) as a 
means of formative assessment? 
3.74 


Scale: 1 = Never; 2 = A few times this year; 3 = Once or twice a month; 


Scale 


None of the differences were statistically significant. 


The remaining districts reported incorporating varying levels 
of the CIF, usually integrating it into other instructional 
initiatives. One district had developed its own instructional 
framework with an accompanying lesson plan structure. The 
staff in this district noted that they were encouraged to use 
the CIF strategies but that it was not required in any way. In 
two of the districts, the CIF work was being supplemented 
with “Teacher Clarity,’ an intervention that helped teachers 
focus on the specific content that students were expected to 
learn. In another school, teachers discussed implementing 
the “Writing to Learn” CIF instructional practice along 

with an effort to do a writing methodology called “I.C.E.” 
(Introduce, Cite, and Explain). A different school had an 
emphasis on differentiation, which integrated some of the 
CIF strategies. 


4 = Once or twice a week; 5 = Almost every day 


The emphasis, or lack thereof, on CIF was driven by the 
amount of attention paid to the practices by school and 
district leadership. In districts with high implementation, 
there were clear instructional expectations and structures 
put in place to support their implementation. In districts 
with perceived lower implementation, leadership may have 
emphasized other instructional approaches or may have 
relied on individual teacher groups to identify instructional 
practices they wanted to implement. 


School staff turnover was also seen as having a negative 
impact on implementation of instructional change. In two 
schools, all instructional emphases were on hold in the final 
year as aresult of changes in principals. In a third school, a 
teacher noted that, even though there had been training on 
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the CIF instructional strategies at the beginning of the 
grant, they were not really bringing new teachers up to 
speed: "I don't see a sustained training effort. There tends 
to be a, "Oh, we've already trained on that," but we didn't 
have all that much training last year when we had some 
new teachers, and we haven't had any this year, and we 
have some new teachers.” 


V.1.4. Strong Postsecondary Partnership 

A strong postsecondary partnership was a critical part of 
ensuring that a Career and College- Ready Academic Program 
was in place. Each district’s primary partnership in this grant 
was with Columbus State, although some districts also had 
partnerships with other postsecondary institutions. 


The school survey included questions that asked school staff 
to comment on specific aspects of their school’s partnership 
with Columbus State. As not all staff members in the project 
schools had awareness of the partnership with Columbus 
State, survey respondents could reply “Don’t know” to any of 
the items within this section of the survey. As shown in Table 
V-3, there was a statistically significant increase in reported 
implementation of most aspects of the partnership from Year 
1 to Year 4. 


Interviews with school and district staff suggested that 
the Columbus State partnership was perceived as strong 
overall by the end of the project. One district coordinator 
commented, “They're truly there to help us 24/7.” Another 


Table V-3. Postsecondary Partnership with Columbus State—Year 1 to Year 4 


Our students have access to college resources and 
facilities at CSCC. 


We have a strong partnership between our school and 
Columbus State Community College (CSCC). 


CSCC is supportive of our school and its vision. 


Our school has a system to provide support to high school 
students taking college courses. 


My school and CSCC collaborate to provide support 
to students in college courses. 


My school and CSCC collaborate to provide support to 
students to graduate from high school. 


lam aware of supports that CSCC provides to high school 
students taking college courses. 


Our school is aware of activities 
and events at CSCC. 


Overall Mean 


Scale 


*Statistically significant difference between Year 1 and Year 4, p<.05. 
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Year 1 Mean | Year 4 Mean 


Year 4 
Percent Agree / 


>To) ole) aulela Me) j 
Don’t Know & 
Ly Cools X=5-J oLo) a hf) 


Strongly Agree 


24.7% 
4.53 4.79* 66% 
(n= 167) 
22.7% 
4.34 4.54 57% 
(n= 153) 
33.3% 
4.25 4.47 54% 
(n= 225) 
19.9% 
4.26 4.45 55% 
(n= 134) 
31.3% 
4.04 4.36* 54% 
(n= 211) 
33.3% 
3.93 4.31* 52% 
(n= 225) 
26.7% 
3.77 4.13* 47% 
(n= 180) 
28.7% 
3.48 3.82* 36% 
(n= 194) 
4.08 4.38* 


1 = Strongly disagree; 2 = Disagree; 3 = Somewhat disagree; 


4 = Somewhat agree;5 = Agree; 6 = Strongly Agree; Dk = Don’t Know 
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coordinator said, “I think they do a phenomenal job of 
supporting us at the school level. I cannot tell you the number 
of times I've met or called [the dual enrollment coordinator], 
hundreds of times.” A third noted, 


“[The partnership is as] strong as it's 
always been. | think it hasn't diminished. 
As a matter of fact, we're currently 
working with [CSCC] on several 
programming opportunities. And | will 
say, whenever there's a phone call, we 
get a phone call back. Whenever there's a 
request from us to meet with them, we'll 
get a meeting.” 


—Coordinator 


A principal in another district acknowledged the support that 
Columbus State had given them with their pathways: 


“The intentionality with that partnership 
has helped us quite a bit in establishing 
pathways here at this school to provide 
more opportunities for our kiddos ... That 
partnership is very, very important to us as 
[it is] our strongest partnership. ... So, we 
need to lean on them.” 


—Principal 


School staff did also recognize a few challenges with the 
partnership. One principal noted struggles with getting 
teachers credentialed to teach college courses. A coordinator 
in another district described how there were often challenges 
with scheduling courses. 


V.2. CAREER AND COLLEGE HEADSTART 

The second Early College Design Principle, Career and 
College Headstart, focused on ensuring that students 
were prepared for college courses and were provided with 
early exposure to the culture and norms of college as well 
as to careers. 


Acore part of readiness involved preparing students 
academically for the dual-enrollment courses that they 
would encounter. The instructional strategies and the COLS 
1101 course described in Section II were two approaches to 
this. Many of the schools also used CCRE funds to purchase 
math and reading software to assess students’ performance 
and provide accompanying differentiated instruction. 
Achieve3000 was the most commonly used literacy software 
and ALEKS was the most common math software. One school 
assessed student performance using the ACCUPLACER 
college placement exam for reading and writing and the 
ALEKS software for math. Based on students’ scores, if 
needed, students would be enrolled in a technical writing 
course and/or a developmental math course that was 
intended to improve their scores and help them avoid the 
need for remediation in college. 


In addition to academic content, the schools also increased 
their focus on college readiness behaviors, otherwise known 
as the “soft skills” needed for success in the postsecondary 
environment. Survey results showed that there was an 
increase in the extent to which teachers incorporated 

these skills in their instruction (Table V-4 page 57). By 

the end of the project, teachers reported statistically 
significantly higher levels of instruction in time management 


and self-advocacy. 


Interviews provided more detail about how schools 
incorporated college readiness skills. In three of the six 
schools we visited, this was done during structured advising. 
During advising periods, ninth and tenth grade students 
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Table V-4. Frequency of College Skills Instruction—Year 1 to Year 4 


College Readiness Skill 


Time Management 
Note Taking 


Organizational Skills 


Advocating for themselves with high school 
and college faculty 


Overall Mean 


Scale 


*Statistically significant difference between Year 1 and Year 4, p<.05. 


would work on interest and career exploration. In eleventh 
and twelfth grades, the focus was on student supports 
related to College Headstart, including college exploration 
and then college applications. In one of these schools, the 
counselors worked together to develop a curriculum to 
provide instruction in soft skills during an advisory period. 
A counselor explained: “We plan it out throughout the 

year, each month has a different theme [communication, 
collaboration, timelines]. So, if we want to talk about 
timeliness, or things of that nature, we would form a lesson 
for each week for each of the grades to go over those things. 
[Counseling staff] definitely meet up every month to discuss 
those things that we want to teach the students.” 


Schools also implemented activities intended to expose 
students to colleges and careers. This included opportunities 
such as college visits or college or career fairs. One 

school hosted “Future Fridays,’ during which business 
representatives would come to classes to talk with students 
about job opportunities, the training that would be required, 
and the benefits that would be offered. Many of the schools 
used Naviance, a career guidance software, to help students 
develop more of a future orientation, lay out their goals, 

and develop a plan for meeting those goals. As a school staff 


member said, 
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3.36 3.58* 
3.23 3.35 
3.58 3.68 
317 3.53% 
3.34 3.54% 


1- Never; 2- A few times this year; 3- Once or twice a month; 
4- Once or twice a week; 5- Almost every day 


“Definitely throughout the time that I've 
been here I've just learned that exposure 
matters with the students. Exposure with 
Future Fridays and careers, exposure with 
college visits, and college fairs, and just 
all-around college programs that we have 
here. Exposure matters, because it does 
let them know that they can attain it, so it 
definitely helps that they are seeing these 
things on a daily and monthly basis. So 
that's one of the biggest things that I've 
learned.” 


—Staff 


More information about how these schools created a future 
orientation can be found in the accompanying policy brief, 
Developing a Future Orientation: How CCRE Schools Focused 
on Preparing Students for Life After High School. 


V.3. WRAPAROUND STUDENT SUPPORTS 
Activities for the Wraparound Student Supports Design 
Principle included offering comprehensive academic 
supports, social and emotional programming and support, 
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and assistance with college applications and financial 

aid. Supports took the form of developing and sustaining 
relationships with students, offering academic assistance 
outside of regular class time, and employing systems that 
identify student needs and suggest targeted interventions. 
Logistic supports, such as registering for placement tests and 
courses, navigating college procedures, and understanding 
how to make use of college resources, were also included in 
the Wraparound Student Supports Design Principle. In this 
section, we focus on supports provided to student taking 
college courses and the general supports provided at the 
high school. 


V.3.1. Supports for College Courses 

As schools expanded their college course offerings, they also 
needed to think about how to provide supports to students 
taking college courses. The fact that the project was being 
led by a community college meant that the supports could 

be more seamlessly coordinated between the school and 

the College than is often the case with dual enrollment 
partnerships. One of the key support strategies was the use 
of the College’s Starfish early alert and monitoring system. 
This system used key indicators (e.g., attendance, assignment 
grades) to identify students who might need assistance in 

a course. Faculty entered data into the system and when 
students missed a pre-specified threshold, such as number 

of days absent or assignments missed, the system usually 
triggered an alert that was sent, via e-mail, to the student and 
the instructor. Faculty were also periodically sent a survey 
asking them to comment on students’ progress in the course 
so those faculty who were not regularly using Starfish could 
also provide input on how their students are doing. 


As of the third year of the project, all schools had identified 
an early alert contact, often a counselor, who also received 
notifications from the early alert system. A Columbus State 
staff person described how the system operated: 
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“TThe schools] have an early alert contact 
that is supposed to monitor Starfish. 
They have access to it, to keep track of 
their students, in addition to our advisors 
monitoring it on our end as well. And 

so, our advisors, when they see there's a 
flag that has been raised, they email the 
student and cc the counselor [regarding] 
that specific concern. Among our 
expectations is that they do that within 
seven days of that flag being raised.” 


—Staff 


Over the course of the project, high school teachers serving 
as adjunct or facilitating faculty became more and more 
accustomed to entering data into the Starfish system. 
According to the data from Columbus State, 96% of the 
college courses taught at the high school were using the early 
alert system in the spring of Year 4. Interviewees who used 
Starfish believed it could provide useful data to high school 
staff, as described by a counselor: 


“I get an email and, if | log into Starfish, 
I can also view all of those alerts. For 
example, right now I got an alert from 
an English professor for a student that 
[had a] participation concern, attendance 
concern, class completion concern, and 
all for the same student. That student is 
basically hospitalized...So, knowing that 
she’s struggling in the class and she 
might not be able to attend class for a 
while, | reached out to [the Columbus 
State advisor] today to see what her 
options were. So, if we have a medical 
note, we can drop the class without 
penalty of a W”. 


—Counselor 
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High school staff turnover did pose a continuity challenge 

for the use of Starfish, however. For example, in Year 4, 

one counselor noted that they no longer had access to 

the system because the previous early alert contact had 
resigned. Schools were expected to identify new contacts who 
Columbus State would then train. 


In addition to Starfish, all students taking dual-enrollment 
courses were eligible to receive tutoring from Columbus State 
both on the college campus and through the online service, 
NetTutor. According to data from Columbus State, 186 
students participated in college tutoring activities in the fall 
of 2019 (Year 4). 


V.3.2. High School Supports 


Over the course of the project, participating schools were 
expected to implement a variety of strategies to improve 
the academic and affective supports for students. As noted 
earlier, many schools utilized software to build students’ 
English and math skills. Other key strategies that were a 
focus of the project included: starting new advisory periods 
or further refining existing advisories, implementing 
indicator and tiered support systems, bringing in additional 
counseling help, and providing explicit support around 
college logistics and financial aid. Survey results showed 


Table V-5. Percentage of Schools Indicating Specific Levels of High School Students’ Receipt of Academic 


and Social/Emotional Supports—Year 1 to Year 4 


Academic and Social/Emotional Supports 


Year 1 Mean Year 4 Mean 


(n=15) (aba I<) 
Advising on courses to take to prepare for college 4.60 4.27 
Advising on choosing college classes 3.80 3.73 
College exam preparation (Test-taking skills for SAT, PSAT, ACT, ACCUPLACER, 240 427" 
or other college placement exams) 
Registration for college exams (SAT, PSAT, ACT, ACCUPLACER, 4.20 433 
or other similar exams) 
Assistance with applications for accessing College Credit Plus courses. 3.40 3.40 
Advising parents about college admissions and financial aid 3.47 4.00 
Helping students through the college admissions process 3.73 4.07 
Helping students through the financial aid process 3.47 4.20* 
Academic tutoring connected to a specific class 3.53 3.33 
Small-group and individualized instruction 2.80 3.00 
Sessions or classes to help students cope with social or emotional issues 2.33 2.93 
Structures to build student-to-student relationships (e.g. peer connections, 527 249 
mentoring). 
Overall Mean 3.42 3.67 


Scale 


*Statistically significant difference between Year 1 and Year 4, p<.05. 


1- Not offered; 2- Fewer than 25%; 3- 25% to 50% 
4- 51% to 75%; 5- Greater than 75% 
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higher reported levels of support in Year 4 (compared to 

Year 1) in several areas, with statistically significant changes 
related to logistical support on financial aid and college exam 
preparation (see Table V-5 page 59). It is important to note 
that this analysis was conducted at the school level, so the 
sample size is small. 


Some schools did implement substantial changes in their 
support systems, described below. 


Advisory Periods. 'Three of the six visited schools had 
advisory periods, which were specific times set aside during 
the school day for school staff to check in with students, 

for students to get tutoring, and/or to provide students 
explicit instruction in college readiness skills, as mentioned 
earlier. For example, one school had a 45-minute advisory 
period that they restructured in Year 3 of the grant to be 
more focused on college and career readiness. Students in 
this school commented that they “definitely noticed a focus 
on college and career readiness. Especially with like [the 
advisory]. It's definitely helped.” Students described that 
“in the [advisory], we plan everything, whether it's college, 
jobs, after high school, graduating, anything...” They said 
that college professors came to talk to them, and they got 
help with signing up for ACTs, writing college essays, time 
management, and knowing college expectations. 


Indicator and Tiered Support Systems. One of the goals of 
the project was to implement systems that would help 
identify students who are ready for college and careers and 
enable schools to intervene if needed. All of the districts 
had some sort of data system to identify students who were 
struggling, but, by the end of the grant, only one of those 
systems had incorporated career and college readiness 
indicators (see box). 


Even though only one of the schools had implemented a 
formal system with the college readiness indicators, all of the 


College Readiness Early Alert 
in South-Western 


Prior to the CCRE project, South-Western’s 


arly alert system involving synthesizing data 
from five or six different data sources into 


an Excel soreadsheet. Then, according toa 
district staff member, “We'd have to do all the 
voodoo that It required to kind of make sense 
of it.” 


About two years in to the project, they 
improved their system by using Tableau 

to integrate the data and create a series of 
dashboards that would allow the district to 
identify students who were at risk or who 
needed acceleration. With the impetus 

of the CCRE project, South-Western also 
incorporated college and career readiness 


indicators, taken from research by David 
Conley and the Bill & Melinda Gates 
Foundation. According to district staff, “The 


dashboards are broken down by grade 

level seven through 12, and they each have 
different components, some overlapping ... 
of college career readiness indicators.” The 


dashboards can help school staff look at a 
variety of indicators, such as students who 


might be earning A's and B's in AP courses 
but only getting al or 2 onthe exam. 


The district has provided training to building 
leadership teams on how to look at and use 
the data. 


Additional Counseling Capacity. Schools recognized that the 
CCRE project required support beyond what their normal 


site visit schools reported an increased emphasis on applying counseling staff could provide. As a result, most of the schools 


data from their systems through Multi-Tiered Systems of in the project expanded their counseling capacity by working 


Support or Response to Intervention. with partner organizations, such as I Know I Can (IKIC) and 
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Communities in Schools (CIS). In some cases, the districts 
used grant funds to support these services and, in other 
cases, they were able to leverage other funding sources. The 
focus of IKIC counselors, in particular, was closely related to 
the CCRE goals of improving college and career readiness. 
These counselors worked with students one-on-one to 
advise, provide college application support, schedule college 
visits, and administer college placement tests. They used a 
curriculum that emphasized exploration for freshmen and 
sophomores, career and college opportunities for juniors, 
and helping seniors apply to colleges. One IKIC counselor 
described typical interactions with students: 


“The majority of our time is spent doing 
one-on-one interactions with students. 
So, either they'll come down here and just 
pop in and say, "Hey, | need help with this." 
| also try to pull about 10 students a day 
to meet with and it depends on what I'm 
kind of looking for that day for students, 
maybe it's just seniors | have not met with 
at all yet, and maybe it's around a certain 
college—you're all interested in Otterbein, 
let's sit down together as a group and 

talk about what that means. We organize 
the rep visits for colleges to come in and 
talk to students. We try to sit in on some 
of those, but then we do some group 
work. So, a couple weeks ago, and then 
next [again] week, we're going to do the 
second round, but I'll go in with our senior 
counselor to do those.” 


—Counselor 


In some schools, these IKIC counselors were responsible 

for guiding much of the college readiness work. Over the 
course of our site visits, we observed increasing coordination 
between school counselors and IKIC counselors in several 
schools. In these schools, IKIC offices were situated close to 


the guidance counselor offices and fully integrated into the 
daily activities of the building. School counselors and IKIC 
counselors both reported working seamlessly with each other 
in these schools. 


CIS counselors were also in several CCRE schools and 
provided a broader range of supports to a more targeted 
population, using a three-tier system to facilitate student 
success. Many students received only Tier I support which 
consisted of “light touches” to monitor those who might need 
consistent but minimal guidance to continue to be successful. 
In many cases, this regular contact was handled by teachers. 
For students who needed Tier II support, CIS counselors 
provided small group interventions; Tier III was more of a 
hands-on approach, with students having daily contact with 
the CIS counselor, who would help monitor progress and 
recognize their successes. We observed students visiting their 
school’s CIS office to touch base, grab a snack, or share a 
success story. 


Support for College Application and Financial Aid Logistics. 
On the survey, the two support indicators that showed 
statistically significant improvement were preparation for 
the ACT/SAT exams and helping students through the 
financial aid process. Helping students through the college 
application process also improved although the change 

over time was not statistically significant. Much of this 
support was probably being provided by the IKIC counselors. 
For example, schools hosted FAFSA nights, often one or 
more evenings where volunteers (usually school and IKIC 
counselors) were available to help parents fill out the FAFSA 
form for their students. 


V.4. SCHOOL-LEVEL 

ORGANIZATIONAL PRACTICES 

The final Early College Design Principle encompassed 
school-level practices that need to be in place to ensure 
effective implementation of the other Design Principles. 

As shown in the logic model (Figure I-1), these practices 
included: 1) structures to support personalized relationships, 
2) establishment of a college-going culture, 3) ongoing job- 
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embedded PD, 4) data-based decision-making, and 5) time 
and support for teacher collaboration. We collected data 
concerning implementation of these different organizational 
practices through the implementation survey and interviews. 
Before discussing the components explicitly mentioned in the 
conceptualization of Design Principles, we begin with a brief 
discussion of school leadership. We also present a summary 
of the survey results for relevant scales; these results and 
additional detail on some scales are included in the 
discussion below. 


Table V-6. School Organizational 
Practices—Year 1 to Year 4 


Leadership 4.61 4.65 
Structures to Support Personalized 

P ‘ 4.28 4.38 
Relationships 
College-going Culture 41 4.21 
Use of High School Data 3.51 3.53 
Use of College Data 1.87 2.06* 
Collaborative Planning 3.64 415 
Job-embedded Professional 

2.36 2.36 


Development 


*Statistically significant difference between Year 1 and Year 4, p<.05. 


V.4.1. School Leadership 

Research conducted by SERVE Center on other reform 
efforts has indicated that effective leadership is a key 
support for school-level implementation. Because of the 
critical role of school leaders, a Key Component in the CCRE 
model was leadership coaching and technical assistance, 
which was designed to improve the practices of leadership 
teams in participating schools. Given this emphasis, the 
evaluation captured perceptions of the quality of school 
leadership through the staff survey and in interviews. The 
survey measured no changes in perceived implementation of 
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leadership skills over time, which was not surprising given 
that there was leadership turnover in many of the schools 
and districts over the life of the project. 


Interviewed staff reinforced the idea that leadership, at both 
the school and district level, was critical to the success of 

the project. In reflecting on lessons learned from the grant, 

a district staff member noted, “First and foremost, broad 
sweeping shifts can only be done successfully if you have 
commitment from decision makers in the district, those that 
have influence and those that have the positional authority to 
make decisions and be a part of it.” 


Although district support was seen as important, the 
principal was also seen as playing a critical role in 
implementation. In a different district, a staff person noted 
the importance of a principal: 


“We have a purpose and that purpose is 
making sure that we're doing everything 
we can for enrollment, enlistment, and 
employment. And so, having the principal 
completely devoted to it, committed to it, 
that's been the big change... when we had 
somebody in leadership in place that was 
not just following the design but believed 
in it and communicated it clearly.” 


—Staff 


A key role for the principal was communicating the 
importance of the project. As one teacher noted, “I really 
think that the principal's role ... is to set the message. I 
mean, he's not in charge of the mechanics of the program, 
but to make sure that we keep that at the forefront of 
what we're thinking about, that that's where we're all 
going.” Additionally, the principal could lead the work by 
coordinating professional development or by providing 
other resources. 
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Leadership did not only reside with the principal. Staff in 
three of the schools described Building Leadership Teams, 
in which teachers (usually the department chairs) worked 
with the administration to assist in rolling out initiatives. 
Additionally, as described earlier under instructional 
practices, the department chairs were sometimes given the 
responsibility of ensuring that instructional changes were 
being implemented. 


Turnover among both school and district leadership was 
generally seen as slowing down implementation. For example, 
a Columbus State staff member noted that principals in 

one district had been struggling with implementation 
because of district leadership turnover: “I think they've been 
stymied over the last year and a half because of changes and 
transition in superintendents.” Staff in another district noted 
how priorities had shifted with a change in school leadership, 
and staff in a third district noted that a new principal could 


put implementation of project-related activities on a hiatus. 


V.4.2. Structures to Support 

Personalized Relationships 

School-Wide Organizational Practices also included 
structures or actions to support personalized relationships 
in which staff knew students and their families well, 
demonstrated a belief that all students can succeed, showed 
respect for and received respect from students, and provided 
mentoring and advising to students. The survey showed no 
statistically significant changes on specific questions that 
asked about personalized relationships. 


As seen in the site visits, most schools did have structures 
that were designed to provide supports for students (e.g., 
MTSS or RTI), which have been described in the sections 
on early alert systems and student supports. The advisory 
periods, implemented by schools in two districts, were 
specifically designed to allow time for teachers to work in 
small groups with students and get to know them better. 
Staff in both of those districts noted that they were trying 
to improve their data tracking in each of those advisories. 


For example, one school introduced digital portfolios for 
students, which allowed for staff to provide more 
personalized supports. 


Schools also reported reaching out to parents through 
evening activities such as college application nights or 
FAFSA nights, but the staff frequently reported low 
attendance at most of these functions. 


V.4.3. College-Going Culture 

In addition to structures to support relationships, the Design 
Principle of School-Wide Organizational Practices focused 
on the establishment of a college-going culture. Evidence 

of a college-going culture included: 1) the demonstration of 
high expectations for students, 2) student beliefs that they 
could succeed in a college environment, and 3) a school-wide 
focus on the academic and social skills necessary for success 
in college. School staff members responded to a set of survey 
items dealing with beliefs, expectations, and supports for 
students to attend college. Table V-7 (page 64) shows the 
differences across the four years trended upwards, but that 
none of the differences were statistically significant. The lack 
of a statistically significant impact was interesting given that 
the other scales that showed positive impacts were related 
to college partnerships, college readiness skills, and college- 
related supports. 


The site visit data indicated that all schools were engaged in 
activities designed to support a college-going culture such as 
student visits to colleges, college representative visits to high 
schools, and evening informational sessions about college 
applications or the FAFSA. 


Many of the schools noted that there was not a strong 
emphasis on college in their community or that many of their 
students may not end up attending college at all. As a result, 
they believed that there needed to be a culture that promoted 
multiple options for students after high school, not just 
college-going. As a teacher in one district said, 
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“I guess one thing is a focus on career as 
well. We have 60% of students on free 

or reduced-price lunch. As a district, as a 
building, we have to look at it. There are 
some kids where college may not be an 
option for them outside of the walls of 
CCP. We know that some students, if they 
have even some college, they have more 
career potential than none. If they know 
they can succeed, then all of a sudden, 
they might work and find resources 
themselves as young adults. There is 
more of the trying; how do we make them 
actually career-ready too?” 


—Teacher 


A different district was focused on making sure each student 
was thinking about the “Three E’s” or what they were 

going to do to make sure they were “Enrolled, Employed, or 
Enlisted” after high school. As a staff member noted, 


Table V-7. College-Going Culture—Year 1 to Year 4 


“Not only are we focusing on the enrolled 
piece with the college credit ... but we're 
also looking at other things that we can do, 
especially with the employee piece. How 
can we get our students credentialed? What 
opportunities can we offer here at the high 
school for those students that we know 
aren't going to go that traditional college 
route, that may need their certificate or the 
trade? And then obviously the enlistment 
piece, when we talk about, what are the 
opportunities, bringing in more recruiters 
.. SO, | think originally, it started with that 
push for the college credit. ” 


—Staff 


Across all visited schools, however, there was a sense that 
the schools were focused on making sure that students were 
planning for life after high school. 


Year 4 
Percent 


Agree / 
Strongly Agree 


The faculty and staff in this school expect every student to have the 


opportunity to receive postsecondary education or training. 


All faculty and staff in this school believe that, if given enough 
support, all students can successfully complete college 
preparatory courses. 


The faculty and staff at the school explicitly and purposefully 
focus on postsecondary aspirations. 


The faculty and staff at the school focus on specific activities that 


lead to enrollment in a postsecondary institution. 


The vision of this school is tied to preparing every student for 
postsecondary education without remediation. 


The school provides activities designed to get all students 
to think of themselves as students who can succeed in a 
postsecondary institution. 


Overall Mean 


Scale 


4.35 4.47 56% 
3.94 3.99 39% 
4.07 4.22 45% 
4N 4.31 46% 
4N 4.12 43% 
4.07 4.16 42% 
41 4.21 


1= Strongly disagree; 2 = Disagree; 3 = Somewhat disagree; 
4 = Somewhat agree; 5 = Agree; 6 = Strongly Agree 
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V.4.4. Ongoing, Job-Embedded Professional 
Development 

Organizational practices also included supports for teachers 
such as ongoing, job-embedded professional development 

to help in implementing the Design Principles. Reponses to 
the survey showed no changes in the frequency of different 
professional development activities. At the school level, job- 
embedded professional development primarily took place 
through teacher collaborative structures (e.g., Teacher-based 
Teams, which are described in more depth in the section 

on teacher collaboration). At least three schools reported 
additional learning teams that often focused on specific 
content areas, which are also described in more depth below. 


One school provided intensive ongoing school-based PD on 
the CIF strategies. The principal reported emphasizing the 
CIF in every half-day PD they had as well as at staff retreats, 


noting, “the structure of the PD being consistently based 
around that one [CIF] topic of instruction.” One teacher 
commented that the in-house PD was better than external 
PD they had received, saying, “I feel like what we've gotten 
within the school, within the district, has been better than 
from the outside.” 


V.4.5. Data-Based Decision-Making 

The School-Level Organizational Practices Design Principle 
also emphasized data-based decision-making in which staff 
members collected and used data about student progress to 
inform discussions about how to reach the goal of students 
being college-ready. On the survey, we asked about the use 
of both high school data and data from college courses. The 
survey showed a statistically significant increase in the use 
of data to improve students’ outcomes in both high school 
and college courses (‘Table V-8). 


Table V-8. Collective Use of High School and College Student Data—Year 1 to Year 4 


Year 1 Mean 


| (ev P4) 


High school staff used data on 
student performance in high 
school classes to improve 
curriculum and instruction. 


Our school collected or received 


data on student performance in 1.76 2.01* 


college classes. 


High school staff used data on 
student performance in college 
classes to improve curriculum 
and instruction. 


High school staff used data 
on student performance in 
college classes to modify 
supports for students. 


Overall Mean 1.87 2.06* 


*Statistically significant difference between Year 1 and Year 4, p<.05. 
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Year 4 Mean 
(n2613) 


2.37 2.49* 


1.69 1.88* 


1.65 1.87* 


Naver A few times (eo) ateor= Mog fo} ater= Mog 
this year twice a month | twice a week 
21% 27% 34% 18% 
36% 36% 20% 9% 
46% 27% 19% 8% 
46% 27% 20% 7% 
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In the interviews, all schools reported using data regularly 
at their schools. A project staff member believed that schools 
were having more data-focused discussions than ever before 
as a result of the project: 


“They're having more of those 
conversations than they have ever had, 
across the board. ... Again, we're moving 
into now, instead of, "Okay, what do we 
want to do as a building? What do we 
want to do overall for our classroom?" 
this is like, "What are we doing for each 
kid? And how are we helping the kids 
understand what is necessary for them to 
be able to matriculate forward with regard 
to the 90% goal?” 


—Staff 


A principal agreed with this assessment, noting, “This year, 
[data use is] hyper-focused. It's more being intentional about 
what we're doing in our teams, what data we're looking at, 
and how we're sharing it. But, I'm very confident that the 
[CCRE program] is the origin of that.” 


A lot of the data use came from districts’ efforts to build 
their RTI or MTSS systems (described in more depth under 
student supports). One district, South-Western, established 
a well-functioning data system that allowed them to track 
student performance on a myriad of indicators. As an 
administrator said, “This is real time, which is nice. So, if I 
go in, I can go, my data center, data visualization Tableau, I 
have attendance dashboards, I have college career readiness 
dashboards.” At least two other districts have worked with 
South-Western to adapt similar approaches to data use in 
their district. 


Districts that were using ALEKS or Achieve3000, which 
were often funded as part of the grant, also noted how those 


programs were providing them with useful data to improve 


student performance. An administrator described how those 
data were used in the school: 


ALEKS and Achieve [3000] really serve 

as our screeners and our progress 
monitoring tools. ... They use that data to 
screen students, to monitor the progress, 
and then make decisions about how to 
intervene with students. But also, those 
programs, especially ALEKS ... they use 
that as a tool to continue the learning for 
their students. And, so, they use that data 
to inform that instruction. 


—Administrator 


In schools with IKIC or CIS counselors, data were also an 
important part of their work. The counselors described how 
they collected and tracked data on key metrics (e.g., the 
percentage of students filling out the FAFSA, the percentage 
of students taking college entrance exams). 


V.4.6. Time and Support for Teacher Collaboration 
Time and support for teacher collaboration served as the 
final component of School-Level Organizational Practices. 
Because the implementation of an early college model was 

a collective, school-wide effort, the opportunity for teachers 
to collaborate served as an important supportive structure. 
We collected data through the survey on whether teachers 
were given more time to collaborate and how frequently 
they collaborated around certain topics. The survey results 
showed administrators reporting a substantial increase in 
time set aside for teacher collaboration although there was 
not a significant increase in teachers’ reported collaboration 
on specific topics. 


The interviews indicated that all school districts had some 
structures to support teacher collaboration. One school had 
a period called “Morning WoW” (Working on the Work) time, 
which was the 45 minutes before students arrived each day. 
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This time was allocated for group staff meetings once a week; 
two days a week were for common planning in their teams, 
and two days a week were for “focus groups” that allowed 
teachers to concentrate on specific topics such as RTI or the 
CIF strategies. 


Instructional rounds, or peer observations, were frequently 
used in one school, whereby the school provided substitute 
teachers so that teachers could visit each other and observe 
teaching practices. A teacher described how they operated: 


“Our class visits are focused on CIF. 
Teachers can, in the beginning say, "Hey, 
I'm going to try something out. | want you 
to give me your feedback on it, or | feel 
like I'm really doing a good job with this 
particular strategy. I’d like you to look for 
that and give feedback there. Teachers 
were identifying, before we went in their 
classrooms, the level of comfort, what 
specifically they were trying. That was 
just our focus. Then our focus shifted 
toward students. They started to use the 
SER, [the] Student Engagement Rubric ... 
[which] focuses on students' behaviors 
and what students are able to do. We were 
looking for evidence of students growing 
and being farther along and different.” 


—Teacher 


Interviewees in two other schools also mentioned peer 
observations. One indicated that they were working well 
but stopped happening when the principal who initiated 
them left. In the second school, staff noted that there were 
some walk-throughs that were done, but they were not 
systematically implemented. 


In addition to adding some opportunities for collaboration, 
the grant was seen by staff in two schools as improving the 
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quality of collaboration and giving more focus to the work. 

In one district, the staff noted that the grant has helped 
them be more intentional in how the different departments 
in the school interact with each other. In another school, the 
principal said, “The push to have a common language, and 
the TBTs, and to make those more effective ... because of [the 
CCRE project] we were able to really have the department 
leads have a clear purpose.” 


V.5. SUMMARIZING SCHOOL IMPLEMENTATION 
As described in this section, the changes that were made 
most broadly across schools were primarily associated with 
college readiness and the expansion of college courses. 
Schools reported increased partnership with Columbus State. 
More students were taking college-level courses and schools 
were expanding supports for these courses, including the 
use of the Starfish Early Alert system and the use of student 
data. Teachers reported increased incorporation of college 
readiness skills into their instruction and staff reported 
having more opportunities for students to prepare for the 
ACT/SAT and more support around financial aid. All of these 
are consistent with creating an environment more focused 
on college. 


In areas that had a less direct connection to college 
coursetaking, school-based changes varied by school and 
district. For example, one district adopted the CIF and 

made district-wide instructional changes. This district also 
incorporated college readiness indicators into their data 
system so that they could identify students who were not 

on track for college or career. Some schools expanded their 
high school supports by establishing or refining advisory 
periods or by bringing on extra counseling staff from external 
organizations like I Know I Can or Communities in Schools. 


According to interviews, the variation in implementation 
came down, as is the case with many initiatives, to the extent 
to which the district and school leadership believed the CCRE 


activities were important. 
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Section VI: Conclusions, Lessons Learned and Recommendations 


The CCRE project was a complex, multi-year initiative that 
used college coursetaking as a lever to move schools to 
develop a culture focused on preparing students not just for 
high school graduation but for what came afterwards—what 
we called a “future orientation.” 


Throughout the grant period, the CCRE partners provided 
arange of implementation supports. They helped schools 
implement college courses and create pathways. They 
provided professional development and coaching to help 
schools change their instruction and improve their early alert 
systems and multi-tiered systems of support. They funded 
efforts to promote college readiness for students through a 
college success course and college readiness software. These 
supports were not always implemented at the level that was 
intended, but this was not because the support for them 
was not provided; it was because not all schools or districts 
availed themselves of these supports. 


The project activities were intended to support schools as 
they implemented Early College Design Principles. The 
evaluation data showed that all schools were shifting their 
focus away from just preparing students for high school 
graduation, to preparing them for the next stage of life, 
whether that be college, a career, or the military. 


The survey and administrative data highlighted changes that 
had been made across all schools. These changes tended to be 
related to college enrollment and success activities. Schools 
showed significant increases in the existence of a common 
vision of instruction. There were more supports focused on 
post-high school readiness and there was increased use of 
data from college courses. 


Implementation tended to vary across districts. In some 
districts, the changes were primarily related to college 
coursetaking. One district, however, used the project as an 
impetus to redesign their entire district improvement plan to 
focus on student-centered instruction and college readiness. 


The analyses of student impacts showed that the support 
activities and changes made by districts did lead to 
improvements in student outcomes. There was a statistically 
significant positive impact on the percentage of students who 
were on-track for high school graduation in ninth and tenth 
grade. There was also a statistically significant increase in 
the percentage of students who were taking college courses, 
although there was not a positive impact on credits earned. 


The information in this report summarizes key 
implementation and impact findings. From these results and 
from other extensive data we have collected over the past five 
years we have identified several lessons and corresponding 
recommendations for practitioners and policymakers seeking 
to do similar work. 


V1.1. THIS WORK REQUIRES STRONG 
PARTNERSHIPS 

The early college model depends heavily on a high-quality 
partnership between school districts and colleges. The CCRE 
project was unique with the work being led by a community 


college, instead of a district or an external organization. 


School, district, and college staff all believed that the 
partnership between Columbus State and the districts 
improved over the course of the grant. This stronger 
partnership allowed the organizations to work through 
many issues that often plague efforts to implement early 
college courses. For example, all districts were able to 
utilize the Starfish Early Alert system to track students’ 
performance in college classes. One district worked through 
its FERPA restrictions to allow college instructors to share 
performance data with other adults in the high school. 
Another school set up its system so students could register 
for courses through their regular high school course 
management system. These kinds of issues only get resolved 
when a postsecondary institution comes to the table willing 
to listen to the high schools and consider how the college 
might operate differently. 
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We found definite advantages to having the work led 

by acommunity college. The College could support the 
streamlining of college policies, make sure that schools had 
access to student data and academic supports, and provide 
support to college and high school adjunct faculty teaching 
college courses. JFF has published a brief, Forging Regional 
Connections: The Role of a Community College in High 
School Transformation, that discusses how a college can lead 
this type of partnership. 


Part of the partnership involved customizing the work. 

The CCRE project was not a one-size-fits-all approach and 
Columbus State worked hard to customize the supports 

they provided to the schools. For example, some districts 
received leadership coaching for their principals while other 
schools preferred to use the resources available for different 
professional development opportunities. The challenge 

with customization is making sure that the core of the 
intervention is not affected. We have published a brief that 
explores the partnership in more detail, focusing particularly 
on the benefits of flexibility and customization, Key to 
Success: Relationship and Adaptability, Lessons Learned about 
Secondary/Postsecondary Partnerships from the College and 
Career Readiness Expansion Project. 


VI.2. THERE IS NO SUCH THING AS 
OVER-COMMUNICATION 

With CCRE, as with other projects we have evaluated, 
participants had concerns with and questions about 
communication at multiple levels. We collected extensive data 
about project awareness, which showed that many teachers 
were unclear about the goals of the project. In interviews, 
staff commented on the desire to know more about what they 
were doing and why they were doing it. 


An additional layer is communication with students. In some 
schools, students reported that they knew about the college 
course opportunities because the information was regularly 
shared at assemblies and other schoolwide events. Other 
students reported hearing very little to nothing about dual 
credit or other college- or career-readiness opportunities. 
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V1.3. EXPANDING ACCESS TO COLLEGE COURSES 
REQUIRES PAYING ATTENTION TO COLLEGE 
READINESS 

CCRE managed to successfully expand access to students 
who are traditionally underrepresented in dual-enrollment 
courses. Expanding access can be done by making sure that 
eligible students are recruited and encouraged to participate. 
However, truly expanding access also means expanding 

the pool of students who are eligible to take college 

courses, which means making sure that more students are 


academically ready for college courses. 


The project schools addressed the issue of readiness in 
several different ways. One district focused heavily on 
improving the quality of instruction to help more students 
be successful in college preparatory courses. Other districts 
used computer software to build students’ reading and 
math skills. Teachers reported integrating more explicit 
instruction on skills such as time management. Our brief on 
expanding access describes some of these activities in more 
depth. 


V1.4. SCHOOLS AND COLLEGES SHOULD THINK 
THROUGH HOW TO SUPPORT STUDENTS TAKING 
COLLEGE COURSES 

Part of expanding access to college courses to more students 
also requires thinking about how to support students 

taking those courses. The CCRE project experimented with 
different approaches to supporting students. One of the more 
successful strategies was to give high school staff access to 
the college’s Starfish system, which identifies students who 
might be struggling in class. High school staff received alerts 
and were able to identify and assist struggling students. 
Columbus State also gave dual enrollment students access to 
its online tutoring program, although it was accessed by only 
a relatively small proportion of students. 


One of the areas where the project struggled was identifying 
an appropriate and sustainable level of advising support 
from the college. Schools often wanted a person there full- 
or half-time, but the college could not sustain that level of 
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funding commitment. This might be an area where partners 
could explore the possibility of jointly funding such positions 
as our previous work has highlighted the importance of 
college liaisons. 


VI.5. EQUITY REQUIRES INTENTIONALITY AND 
PURPOSEFULNESS 

Recent research has highlighted the equity challenges 
present in the current expansion of dual enrollment. 

CCRE has gone against the trend by focusing on expanding 
college coursetaking to students who are traditionally 
underrepresented in college. The work of this project, 

plus the data coming out of early colleges, suggest that 
disparities in access are not inevitable and that an intentional 
focus on equity can ensure that all populations of students 
can participate. 


Part of this focus on equity means ensuring that students 
are ready for college courses and that they are supported 

in college courses, as noted in the sections above. As with 
expanding access, being intentional and purposeful about 
issues of equity in these supports will also go a long way. 


VI.6. CREATING A FUTURE ORIENTATION 
INVOLVES DIFFERENT PATHWAYS FOR 
DIFFERENT STUDENTS 

Even though CCRE was supposed to be focused on both 
college and career readiness, we did see tensions between the 
two goals. In some schools, staff thought there was too much 
emphasis on college readiness at the expense of students who 
might want to go directly into the workforce. Staff in other 
schools believed that the school had too low of expectations 
for their students and having students take college courses 
made other staff more likely to believe that their students 
could be successful in college. 
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This tension is likely to remain in any project that is focusing 
on preparing students for life after high school. We see our 
term “future orientation” as providing a broader frame that 
schools can use, even if operationalizing it will require schools 
to wrestle with the same tensions. Nevertheless, keeping the 
focus on preparing students for options after high school may 
help with the balancing act. 


VI.7. SUSTAINABILITY REQUIRES 
INSTITUTIONALIZATION 

A final insight from CCRE is that sustaining this work 
requires institutionalization of practices and procedures. 
Most people will think about institutionalization at the 
school level, ensuring that practices—college-level courses, 
instructional practices, student supports, data usage, 
teacher collaboration—become part and parcel of the way a 
school does business. This allows these practices to survive 
staff turnover. 


Institutionalization is also important for the colleges that 

are providing the supports to the schools. Columbus State 
has noted that this project resulted in many changes at the 
College. They have streamlined their policies and changed the 
way that they interact with the high schools. These practices 
have also been institutionalized so that they can continue 
supporting this work into the future. 
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APPENDIX A: FIDELITY OF IMPLEMENTATION 


Table A.1. Fidelity of Implementation (FOI) of Key Components 


Key Component 


Key Component 1: 
Management 
structure 


Key Component 2: 
Professional 
development and 
coaching for district 
and school staff 
(provided by CCRE 
partners) 


Tate |fer- keyg 


[Uh alc=-1ale)(e)] 


Participation in regional 
CCRE Cabinet 

(District expected to 
participate in 8 meetings in 
Year 2, 3 meetings in Year 3) 


Participation in one-on-one 
district meetings 

(District expected to 
participate in 3 meetings) 


Participation in 

Core Team meetings 
(District expected to 
participate in 80% of twice- 
monthly meetings.) 


Professional development for 
district staff 


(Districts expected to 
participate in more than 48 
hours in Year 2 and 12 hours in 
Year 3) 


Professional development 
for school leadership 


(Schools expected to 
participate in more than 48 
hours in Year 2 and 32 hours 
in Year 3) 


Professional development for 
school staff 


(Schools expected to 
participate in at least 30 hours 
in both years) 


a'(-¥-] ad oO) DY -S-tol df old fey al 
exer yAirs)) 


3 of 7 districts had 
representation at 8 CCRE 
Cabinet meetings. District 
participation ranged from 3 to 
8 meetings with an average of 
6.4 meetings per district. 


One-on-one meetings 
were not implemented 
until Year 3. 


FOI was not tracked for this 
indicator in Year 2. 


5 of 7 districts met the 
threshold of 48 or greater 
district staff PD hours in 2017- 
18. The number of PD hours 
by district ranged from 12 to 91 
with an average of 55.0 hours 
per district. 


2 of 16 schools met the 
threshold of 48 or more school 
leadership PD hours in 2017-18. 
The number of PD hours by 
school ranged from 11 to 86 
hours, with an average of 40.5 
hours per school. 


13 of the 16 program schools 
met the threshold of 30 school 
staff PD hours in 2017-18. The 
number of PD hours by school 
ranged from 19 to 130 hours 
with an average of 49.2 hours 
per school. 


a'(-¥-] ae oO) DY-S-Lol dl el diel al 
(2018-19) 


7 of 7 districts had 
representation at 2 CCRE 
Cabinet meetings. The 
cancellation of the third 
Cabinet meeting led to no 
districts meeting FOI. 


2 of 7 districts had 3 one-on- 
one meetings, with all districts 
having at least 1 meeting. FOI 
was not met for this indicator. 


No districts met the threshold 
of attendance at 16 meetings 
because only 11 meetings were 
held. Each district attended 

0 to Tl meetings, with an 
average of 5.7 meetings per 
district. 


7 of 7 districts met the 
threshold of 12 or more district 
staff PD hours in 2018-19. 

The number of PD hours by 
district ranged from 35 to 79, 
with an average of 50.0 hours 
per district. 


Tl of 16 schools met the 
threshold of 32 or more school 
leadership PD hours in 2018-19. 
The number of PD hours by 
school ranged from 12 to 63 
hours, with an average of 37.5 
hours per school. 


12 of the 16 program schools 
met the threshold of 30 school 
staff PD hours in 2018-19. The 
number of PD hours by school 
ranged from 18 to 108 hours, 
with an average of 55.5 hours 
per school. 
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Table A.1. Fidelity of Implementation (FOI) of Key Components (continued) 


Tate ler- keyg 


Year 2 FOI Description 


Year 3 FOI Description 


Key Component 


Key Component 2: 
Professional 
development and 
coaching for district 
and school staff 
(provided by CCRE 
partners) 
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Coaching/technical assistance 
for district staff 

(District expected to receive 
technical assistance contacts 
in at least 8 of 9 months in 
Year 3) 


Coaching/technical assistance 
for school leadership 

(School leaders expected to 
receive technical assistance 
contacts in at least 8 of 9 
months in Year 3) 


Coaching/ technical assistance 
for school staff 

(School staff expected to 
receive technical assistance 
contacts in at least 8 of 9 
months in Year 3) 


(2017-18) 


Discussion with program staff 
revealed that the coaching 
for district staff indicator was 
better framed as “technical 
assistance” for districts. 
Program records showed 
multiple touchpoints of 
technical assistance from 
grant partners to all 7 districts 
in 2017-18. 


Tl of 16 schools met FOI for 
leadership coaching visits in 
2017-18. Principals at program 
schools received a range of 2 
to 15 leadership coaching visits 
with an average of 9.3 visits 
per school. 


8 of 16 schools met FOI 

for instructional coaching 
visits in 2017-18. Schools 
received a range of 0 to 12 
instructional coaching visits 
with an average of 8.1 visits 
per school. 


(2018-19) 


4 of 7 districts met the 
threshold of technical 
assistance contacts in 8 or 
more months in 2018-19. The 
number of months with one 
or more technical assistance 
contacts ranged from 4 to 9 
months, with an average of 7.4 
months per district. 


6 of 16 schools met the 
threshold of technical 
assistance contacts in 8 or 
more months in 2018-19. The 
number of months with 1 or 
more technical assistance 
contacts ranged from 0 to 9 
months, with an average of 5.3 
months per school. 


5 of 16 schools met the 
threshold of technical 
assistance contacts in 8 or 
more months in 2018-19. The 
number of months with 1 or 
more technical assistance 
contact ranged from 0 to 9 
months, with an average of 3.3 
months per school. 
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Table A.1. Fidelity of Implementation (FOI) of Key Components (continued) 


Key Component 


Tate ler-1kel a 
(Ub elc= ial) (e)] 


Year 2 FOI Description 
exer yAir)) 


Year 3 FOI Description 
(2018-19) 


Key Component 3: 
Curriculum 
development 

and alignment 


Key Component 4: 
Professional 
development for 
college faculty 


Support for integrated 9-14 
pathways (College expected 
to provide support to each 
district) 


Support for work-based 
learning curriculum 

(College expected to provide 
support to each district) 


Credit-bearing college 
success course(College 
develops course and it is 
offered in schools). 


PD for college faculty teaching 
in high schools 

(80% of faculty attend two 
sessions). 


PD for high school teachers 
(volunteer adjunct faculty) 
teaching dual enrollment 
college courses 

(80% of faculty attend two 
sessions) 


Columbus State provided support to each district through 
individual planning sessions and PD events at the College. Project 
records indicate that 7 of 7 districts received support from the 
college for pathways in Years 2 and 3. Thus, this indicator was met. 


Columbus State provided support to districts via individual 
planning sessions. Fidelity was met for this indicator. 


Columbus State developed and refined COLS 1101 for use in 
program schools, meeting FOI. FOI was met as the course was 
offered throughout the CCRE schools. FOI did not require all 
schools to offer the course; in Fall 2018, the course was offered in 

Tl of 16 program schools and in Spring 2019, the course was offered 


in 7 of 16 schools. 


Columbus State identified 54 
instructors delivering courses 
in CCRE districts in 2017-18. 5 
of 54 (9%) faculty members 
attended two or more 
sessions, short of the 80% 
target for this indicator. 


CCRE districts had from 1 to 

25 staff members identified in 
their district to receive PD as 
College Credit Plus adjuncts in 
2017-18. District percentages of 
adjunct faculty ranged from 
0% to 40%, with no district 
meeting fidelity for this 
indicator. 


Columbus State identified 60 
instructors delivering courses 
in CCRE districts in Fall 2018 
and Spring 2019. 12 of 60 (20%) 
faculty members attended 
two or more sessions, short 

of the 80% target for this 
indicator. 


CCRE districts had from 1 to 
28 staff members identified in 
their district to receive PD as 
College Credit Plus adjuncts 
in Fall 2018 and Spring 2019. 
District percentages of 
adjunct faculty attending 2+ 
sessions ranged from 0% to 
43%, with no district meeting 
fidelity for this indicator. 
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Table A.1. Fidelity of Implementation (FOI) of Key Components (continued) 


Key Component 


Tate lfer- keyg 
[Ub elc=-1ale)(e)] 


Year 2 FOI Description 


(2017-18) 


Year 3 FOI Description 
(2018-19) 


Key Component 5: 
Student support 
activities (provided 
by CSCC) 
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Early alert system— 

high school classes 

(Early alert system in place, 
incorporating college and 
career readiness indicators— 
Year 2; system in use—Year 3) 


Early alert system— 
college classes 

(Staff trained on early alert 
system—Year 2; early alert 
system in use—Year 3) 


Academic advising 
(College advisors are in place 
for each district) 


Tutoring 
(College provides tutoring for 
students in each district) 


At the end of the 2017-18 
school year, no schools had 
systems that incorporated the 
required indicators; therefore, 
FOI was not met for this 
indicator in Year 2. 


In Year 2, project records 
indicated that 16 of 16 CCRE 
schools had at least one 
staff member trained on the 
Starfish platform. Thus, this 
indicator met FOI for Year 2. 


Program records indicated 
that advisors from Columbus 
State were assigned to all 
CCRE schools and meeting 
with students on-site during 
the 2017-18 school year. Thus, 
this indicator met FOI for Year 
2. 


In Year 3, the intent was 

that building-level teams 
would meet monthly to 
assess and analyze student 
level data for at-risk students 
in the following areas: 1) 
behavior, 2) academics, and 3) 
attendance. From interviews, 
4 of 7 districts reported having 
RTI teams that met regularly, 
and one of those districts also 
had a centralized data system, 
although the two parts of 

the system were not yet 
connected. FOI was thus 

not met. 


A core set of Columbus State 
staff monitored the system 
and ensured that each high 
school contact received 
regular updates. Each school 
had an individual designated 
to receive Starfish updates. 
86% of college course sections 
in the Fall 2018 and Spring 
2019 semesters used Starfish, 
so FOI was met. 


The expectation for FOI is that 
an advisor be in place and 
provide support to schools 
and/or students. Given that 
each school had access to 

an advisor and that advisors 
visited schools, this FOI 
measure is met. 


The FOI indicator states, “Academic tutoring is provided to 
students identified as at-risk.” All students who participated in 
Columbus State classes had access to NetTutor, the online tutoring 
system, which met the expectations for FOI for both Years 2 and 3. 
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Table A.1. Fidelity of Implementation (FOI) of Key Components (continued) 


Tate lfer- ike) 


Year 2 FOI Description 


Year 3 FOI Description 


Key Component 


Key Component 6: 


District strategic 
implementation 
plan 


Key Component 7: 


Leadership 
development 
(provided by 
school districts) 


Key Component 8: 


Professional 
development 
to school staff 
(provided by 
districts) 


[Ub elc=1ale)(e)] 


District strategic 
implementation plan 
(District creates plan with 
early college principles) 


Coordination and 
communication 
(i3 Coordinator in place) 


Leadership development 
for leaders of each school-- 
provided by district staff 
(District provides 2 days (or 
12 hours) of professional 
development annually). 


PD for school building staff, 
provided by the school district 


(School staff participate in 
more than 12 hours annually) 


(2017-18) 


All 7 districts submitted 
strategic plans in the 2017-18 
school year. Plans contained 
strategies aligned with the 
CCRE Key Components 

and Early College Design 
Principles. 


(2018-19) 


5 of 7 districts submitted 
updated plans to Columbus 
State in Fall 2018. The plans 
were used in one-on-one 
district meetings to guide 
project implementation. As 
not all districts submitted a 
plan, FOI was not met for this 
indicator. 


7 of 7 districts had a coordinator in place to manage program 
activities in both years; thus, the program met FOI. 


10 of 16 schools met FOI for 
hours of district-provided 
leadership development 

in 2017-18. CCRE districts 
provided an average of 

21.4 hours of leadership 
development to schools, 
ranging from O to 39 hours by 
school. 


12 of 16 met FOI for hours of 
district-provided school staff 
PD in 2017-18. CCRE districts 
provided an average of 26.1 
hours of PD to school staff, 
ranging from 0 to 50 hours by 
school. 


8 of 16 schools met FOI for 
hours of district-provided 
leadership development 

in 2018-19. CCRE districts 
provided an average of 

11.6 hours of leadership 
development to schools, 
ranging from O to 27 hours by 
school. 


Tl of 16 met FOI for hours of 
district-provided school staff 
PD in 2018-19. CCRE districts 
provided an average of 14.4 
hours of PD to school staff, 
ranging from O to 28 hours by 
school. 
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Table A.1. Fidelity of Implementation (FOI) of Key Components (continued) 


Key Component 


Tate lfer- keyg 
{Uh alc=1ale)(e))] 


Year 2 FOI Description 
exer yAirs)) 


a'(-¥-] a oO) I DY-S-Lol d | old ela) 
(2018-19) 


Key Component 9: 
Creation of 
integrated 9-14 
pathways with 
work-based 
learning support 
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Integrated 9-14 pathways 
(Pathways developed and 
implemented) 


Integration of work-based 
learning practices in 9-14 
pathways (Pathways include 
work-based learning activities 
as appropriate) 


Tl of 16 schools in 5 of 7 
districts provided evidence of 
developed pathways. 


Program records did not 
indicate the integration of 
work-based learning into 9-14 
pathways. 


7 of 7 districts provided 
documentation of pathways 
that included college 
coursework. For districts with 
multiple high schools, not 

all pathways were available 
at all schools, but there was 
evidence that students in 
each program high school 
had access to at least one 
integrated pathway, which 
indicates that FOI was met for 
this indicator. 


The pathway documentation 
for 4 of 7 districts included 
references to work-based 
learning, particularly 
internships. There is also 
evidence from interviews that 
career awareness activities 
occurred in schools. However, 
the lack of explicit integration 
of work-based learning 

into each district’s pathway 
indicates that the indicator 
was not fully met. 
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APPENDIX B: REPRESENTATIVENESS 


These representativeness tables are intended to demonstrate that the school-level baseline equivalence and the outcome 
analyses were calculated on samples that met WWC’s representativeness criteria. 


Table B-1. Representativeness for the Ninth Grade On-Track Sample 


Number of Number of 9th Grade 
9th Grade Students in | Students in State-Level fof celU) ow Nada ia(ola) 
Tod Yolo) IN’ (=t-1 a Treatment Status P Na) Wa dtomcs-Toal eo) (-) Enrollment File Percentage 
Treatment 3,918 4,873 19.6% 
j 0, 
Baséline Vear Comparison 7,454 9,185 18.8% 
(eoIee) Total 11,372 14,058 19.1% 
Differential Attrition 0.8% 
Treatment 3,923 4,990 21.4% 
j ‘0, 
Duteome ence Comparison 7,701 9,541 19.3% 
OW18) Total 11,624 14,531 20.0% 
Differential Attrition 2.1% 
Treatment 3,830 4,758 19.5% 
Outcome Year 3 Comparison 7,153 8,727 18.0% 
(2018-19) 
Total 10,983 13,485 18.6% 
Differential Attrition 1.5% 


Table B-2. Representativeness for the Tenth Grade On-Track Sample 


Number of Number of 9th Grade 
9th Grade Students in | Students in State-Level Group Attrition 
School Year Treatment Status PN at) Wadtomcy-Taay el (=) Enrollment File Percentage 
Treatment 3,422 4,271 19.9% 
Comparison 6,582 8,036 18.1% 
Outcome Year 3 
Total 10,004 12,307 18.7% 
Differential Attrition 1.8% 


Note: Baseline for this outcome is the 9th grade on-track sample, shown in Table B-1. 
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Table B-3 includes representativeness for both the confirmatory sample (students who met the ADM threshold but may have 
had missing achievement data) and the analytic sample used for the sensitivity analysis (students with no missing data). 


Table B-3. Representativeness for the Dropout Sample 


Number of 
Number of Students in | Students in State-Level (ef celU) oM Nada ia(ola) 
Treatment Status P Na) Wa dtomcs-Toal el (=) Enrollment File Percentage 
Treatment 14,373 15,416 8.4% 
‘i Comparison 28,389 30,999 6.8% 
Baseline Year 
(ores) Total 42,762 46,415 7.9% 
Differential Attrition 1.7% 
Treatment 14,875 16,035 7.2% 
j ‘0, 
Ouieeme Vanes Comparison 27,845 30,449 8.6% 
noe) Total 42,720 46,484 8.1% 
Differential Attrition 1.4% 
Treatment 13,382 16,035 16.5% 
Outcome Year 3 Comparison 25,593 30,449 15.9% 
with No Missing Data 
(2018-19) Total 38,975 46,484 16.2% 
Differential Attrition 0.6% 


@This sample includes all students with missing data and represent the sample used in the confirmatory analysis. 
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Table B-4. Representativeness Summary—College Course Enrollment 


Number of 10th-12th 


Number of 10th -12th (eie-e(-Moj able (-yal eM la) 
Grade Students in State-Level (efcole|oW-Naddiaieya) 
boTod aLoto) IN'(=y- 1g Treatment Status P NaF) Wa atomcs-Toal oe) (=) Enrollment File Percentage 
Treatment 10,603 10,543 -0.6%? 
Comparison 22,161 21,814 -1.6%? 
Baseline Year (2015-16) 
Total 32,764 32,357 -1.3%? 
Differential Attrition 1.0% 
Treatment 10,192 11,277 9.6% 
j 0, 
Gurssrneieace Comparison 20,524 21,722 5.5% 
oe) Total 30,716 32,999 6.9% 
Differential Attrition 4.1% 
Treatment 10,737 11,202 4.2% 
0, 
Guieenevesre Comparison 21,104 21,528 2.0% 
fens 20) Total 31,841 32,730 2.7% 
Differential Attrition 2.2% 


We used the same ADM threshold to identify the sample for baseline measures and for the outcome measures. For the baseline measures, we did not exclude students with 
missing achievement data (although we did for the outcome measures). As a result, the number of students included in our baseline measures actually exceeds the enrollment 
numbers in the ODE data. Attrition for these measures at baseline can be considered to be 0%. 
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Table B-5. Representativeness Summary—College Credits Earned by Graduates 


Number of Students in | Number of Students in 


Outcome 5 2015-16 the Grad Core Sample (efcoleloW-Naddiaieya) 
School Year Treatment Status Baseline Sample from Study Schools Percentage 
Treatment 2,665 2,665 0.0% 
Comparison 6,039 6,039 0.0% 
Baseline Year (2015-16) 
Total 9,704 9,704 0.0% 
Differential Attrition 0.0% 
Treatment 3,060 3,362 9.0% 
j 0, 
Sutesmeveses Comparison 6,424 6,858 6.3% 
Poe) Total 9,484 10,220 7.2% 
Differential Attrition 2.7% 
Treatment 3,122 3,222 5.0% 
j 10, 
Fe ee Comparison 6,615 6,829 5.9% 
Gole-29) Total 9,737 10,051 5.6% 
Differential Attrition 0.9% 


There is no attrition for the baseline year because the sample includes all students in the graduation file. 
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APPENDIX C: SURVEY SCALES 


Table C-1. Survey Scales Summary 


Tor) (3 


Reliability Estimate 


Sample Items L>X-1-1 ofo) al--MO) old (ol at] 


Common 
Instructional 
Framework (CIF) 


a=0.92 


Student 
Enrollment 

in Advanced 
Courses and 
Postsecondary 
Pathways 


a=0.76 


Strong 
Postsecondary 
Partnership 


a= 0.89 


Postsecondary 
Partnership with 
Columbus State 


a= 0.92 


This school year, how frequently have you... 

- Asked students to explain their thinking in class? 

- Had students work together on projects or assignments? 

- Made connections between material covered previously 
and new content? 


This year, what percentage of your students are... 

+ Enrolled in one or more Advanced Placement (AP) courses? 

+ Enrolled in a pathway containing both high school and college 
credit courses? 

+ Ontrack to earn 3+ college credits before high school 
graduation? 


This school year, how many times did you participate in the 

following activities that involved collaboration between the high 

school and CSCC? 

- Meetings to plan the college courses that high school students 
may take. 

- Meetings with college faculty to establish curriculum alignment. 

- Meetings with college faculty to discuss student support. 


Please indicate the degree to which you agree or disagree with 

each of the following statements regarding 

the partnership between your school and Columbus State 

Community College (CSCC). 

- We have a strong partnership between our school and Columbus 
State Community College (CSCC). 

+ Our students have access to college resources and facilities at 
CSCC. 

« Myschool and CSCC collaborate to provide support to students 
in college courses. 


1- Never 

2- A few times this year 
3- Once or twice a month 
4- Once or twice a week 
5- Almost every day 


1- Not offered 

2- Less than 25% 

3- 25% to 50% 

4- 51% to 75% 

5- Greater than 75% 


1- Never 

2- 1-2 times 

3- 3-6 times 

4- More than 6 times 


1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 
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Table C-1. Survey Scales Summary (continued) 


Tor) {=} 


Reliability Estimate Sample Items L>X-1-1 ofo) al--MO@) old (ol at-4 


Student How many students in your school participate in the following 1- Not offered 

Participation activities at some point before they graduate? 2- Fewer than 25% 

in Work-Based - Work study programs. 3- 25% to 50% 

Learning + Career internship or on-the-job learning (such as job shadowing 4- 51% to 75% 

Activities or apprenticeships). 5- Greater than 75% 
+ Take a career assessment survey and review the results with a 

a= 0.62 staff member or mentor. 

Student Please estimate the percentage of students who receive the 1- Not offered 

Participation following services from the school at some point during their high 2- Fewer than 25% 

in College school experience. 3- 25% to 50% 

Headstart + Structured advising on skills 4- 51% to 75% 

Activities students need in college (e.g., notetaking skills, time 5- Greater than 75% 

management, self-advocacy, etc.). 
a=0.77 + Tours of college campuses. 


+ Tutoring or mentoring from current college students. 


College Skills How frequently have most of your students received explicit 1- Never 

Instruction— training in the following areas in standard academic courses, that 2- A few times this year 

Administrators is, NOT in specific college readiness courses. 3- Once or twice a month 
+ Time management. 4- Once or twice a week 

a=0.96 + Note taking. 5- Almost every day 


- Advocating for themselves with high school and college faculty. 


College Skills In the current school year, how frequently have you explicitly 1- Never 

Instruction— taught in the following skills to students in your classes... 2- A few times this year 

Teachers + Time management. 3- Once or twice a month 
- Note taking. 4- Once or twice a week 

a=0.84 - Advocating for themselves with high school and college faculty. 5- Almost every day 
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Table C-1. Survey Scales Summary (continued) 


oTor-) (3 


Reliability Estimate Sample Items L>X-1-1 ofe) al-{-MO) old (ol at] 


Student Receipt Please estimate the percentage of students who receive the 1- Not offered 
of Academic and following services from the school. 2- Fewer than 25% 
Social/Emotional + Advising on courses to take to prepare for college. 3- 25% to 50% 
Support + Academic tutoring connected to a specific class. 4- 51% to 75% 

+ Sessions or classes to help students cope with social or emotional 5- Greater than 75% 
a =0.93 issues. 
Presence Please mark the extent to which you agree or disagree with the 1- Strongly disagree 
of Student following statements about the academic and social supports for 2- Disagree 
Supports students in your school. 3- Somewhat disagree 

- |regularly provide academic supports to students who need it. 4- Somewhat agree 
a=0.84 + Our school has a system to identify students in need of social/ 5- Agree 

emotional support. 6- Strongly agree 


- All students have opportunities to 
get academic support during the school day. 


Perceptions of Please indicate the extent to which you agree or disagree with the 1- Strongly disagree 
Leadership following statements about the leadership team at your school: 2- Disagree 

- Provides effective leadership at this school. 3- Somewhat disagree 
a =0.92 + Monitors instruction on a regular basis. 4- Somewhat agree 

- Actively supports collaboration among staff members. 5- Agree 


6- Strongly agree 


Structures Please mark the extent to which you agree or disagree that the ]1- Strongly disagree 

to Support following statements about the relationships in your school. 2- Disagree 

Personalized - Faculty and staff members respect all the students in this school. 3- Somewhat disagree 

Relationships + Faculty and staff in this school believe that all students can do 4- Somewhat agree 
well. 5- Agree 

a=0.92 - Mentors or advisers regularly meet with students. 6- Strongly agree 
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Table C-1. Survey Scales Summary (continued) 


boTor) (3 


Reliability Estimate 


Sample Items 


L>X-1-1 oLe) al-{-MO) old (ol at-4 


College-Going 
Culture 


a=0.92 


Data-Based 
Decision-Making 
Activities 


a= 0.82 


Understanding 
of CCRE and 
Partnership with 
CSCC 


a= 0.96 
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Please mark the extent to which you agree or disagree with the 

following statements about the college readiness for students in 

your school. 

+ The faculty and staff in this school expect every student to have 
the opportunity to receive postsecondary education or training. 

+ The faculty and staff at the school focus on specific activities that 
lead to enrollment in a postsecondary institution. 

+ The vision of this school is tied to preparing every student for 
postsecondary education without remediation. 


How frequently do you participate in the following activities? 

- Analyze student progress or performance data. 

- Use data to make decisions about modifying instructional 
practices in your classroom or school. 

How frequently have the following activities occurred at your 

school over the past year? 

+ Our school collected or received data on student performance in 
college classes. 

+ High school staff used data on student performance in college 
classes to improve curriculum and instruction. 


Please indicate the extent to which you agree or disagree with 

the following statements about the College and Career Readiness 

Expansion partnership with Columbus State Community College 

funded through the i3 grant regarding shared understanding. 

- |understand what it means to implement CCRE-related college 
and career readiness activities in my school. 

+ Lunderstand the components of the partnership with Columbus 
State Community College for which | am responsible. 

- |understand how the components of support provided through 
our partnership with Columbus State Community College work 
together. 


1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 


1- Never 

2- A few times this year 
3- Once or twice a month 
4- Once or twice a week 


1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 


Tor) (3 


Reliability Estimate 
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Table C-1. Survey Scales Summary (continued) 


Sample Items 


L>X-1-1 ofo) al--MO) old (ol at-y 


Communication 


about Early 
College 
Strategies 


a=0.93 


Beliefs about 
Early College 
Strategies 


a= 0.93 


Action Toward 
Implementing 
Early College 
Strategies 


a= 0.88 


The Career and College Readiness Expansion Program is intended 
to help schools implement Early College strategies in their schools. 
Please indicate the extent to which you agree or disagree with 

the following statements regarding communication about Early 

College strategies: 

+ Our school leadership has clearly articulated the purpose of 
implementing Early College strategies in our school. 

- Program staff members from outside our building have clearly 
articulated the purpose of implementing Early College strategies 
in our school. 

+ The goals of the Early College partnership are frequently 
reinforced by district staff and other representatives supporting 
Early College strategies (e.g., staff from ESC and JFF). 


The Career and College Readiness Expansion Program is intended 

to help schools implement Early College strategies in their schools. 

Please indicate the extent to which you agree or disagree with the 

following statements regarding beliefs and motivation about Early 

College strategies: 

- [strongly value the kinds of changes called for by implementing 
CCRE-related college and career readiness activities in my school. 

- lam personally motivated to make CCRE-related college and 
career readiness activities work in my school. 

+ The leadership team enthusiastically participates in activities 
related to implementing CCRE-related college and career 
readiness activities. 


Please indicate the extent to which you agree or disagree with 

the following statements regarding support and action relative to 

Early College strategies: 

- Lam capable of making the kinds of changes needed to 
implement CCRE-related college and career readiness activities. 

- [have the support | need to implement CCRE-related college and 
career readiness activities in my school. 

+ The leadership team takes specific actions to support CCRE- 
related college and career readiness activities in our school. 


O- Don't know 

1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 


O- Don't know 

1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 


O- Don't know 

1- Strongly disagree 

2- Disagree 

3- Somewhat disagree 
4- Somewhat agree 

5- Agree 

6- Strongly agree 
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Table C-1. Survey Scales Summary (continued) 


Tor) {=} 


Reliability Estimate Sample Items 


Ongoing Job How much professional development have you received in the 
Embedded PD following areas over the past year? 
+ Instructional strategies in your content area. 
+ College and career readiness (e.g., course selection, time 
management, etc.). 


Time and How frequently do you collaborate with other school staff on the 
Support following: 

for Teacher + Lesson or unit planning 

Collaboration - Peer observations & feedback 


- Using research or data to improve instruction 
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O-None 

1- Asingle presentation 

2-Multiple sessions 

3-Multiple sessions with 
on-site follow-up 


1- Never 

2- A few times this year 
3- Once or twice a month 
4- Once or twice a week 
5-Almost every day 
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APPENDIX D: SUPPLEMENTAL ANALYSES 


Table D.1. Subgroup Analysis for Taking Either CCP or AP Courses 


Unadj. Tan) ex=Koas Effect Size 
Population (ej ¢-Ye (=m M-0"(-) |  =s- 1 a) ola nl ol | (ovoy nn) oy (=s-18) Estimate | (Cox’s Index) 


20,929 41,628 
All Students Grades 10-12 29.9% 22.7% +7.2pp** 0.225 0.003 
(16 sch) (32 sch) 


Economically 12,758 24,685 
: Grades 10-12 22.8% 15.1% +7.7pp** 0.307 0.002 
Disadvantaged (16 sch) (32 sch) 
Not Economically 8,171 15,886 
. Grades 10-12 43.4% 33.7% +9.6pp** 0.248 0.008 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 9,928 20,126 
ot, Grades 10-12 21.2% 14.4% +6.8pp** 0.286 0.005 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 11,001 21,458 
_. Grades 10-12 39.2% 30.5% +8.7pp** 0.232 0.004 
Race/Ethnicity (I6sch) | (29 sch) 


13,216 26,638 
All Students Grades 11-12 36.3% 29.1% +7.3pp* 0.201 0.011 
(16 sch) (32 sch) 


Economically 7,777 15,515 
; Grades 11-12 27.4% 18.9% +8.5pp** 0.293 0.006 
Disadvantaged (16 sch) (32 sch) 
Not Economically 5,439 10,453 
3 Grades 11-12 53.4% 43.3% +10.2pp* 0.248 0.010 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 6,162 12,879 
ne Grades 11-12 26.2% 18.2% +8.0pp** 0.284 0.009 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 7,054 13,732 
= Grades 11-12 47.9% 39.3% +8.6pp* 0.213 0.016 
Race/Ethnicity (I6sch) (29 sch) 


7,713 14,990 
All Students Grade 10 18.8% 11.4% +7.4pp** 0.356 0.003 
(16 sch) (32 sch) 


Economically 4,981 9,170 
. Grade 10 15.5% 8.8% +6.7pp** 0.391 0.004 
Disadvantaged (16 sch) (32 sch) 
Not Economically 2,732 5,433 
; Grade 10 25.0% 15.4% +9.6pp* 0.366 0.028 
Disadvantaged (12 sch) (19 sch) 
Underrepresented Race/ 3,766 7,247 
aed Grade 10 13.8% 7.7% +6.2pp** 0.399 0.007 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 3,947 7,726 
os Grade 10 24.7% 14.9% +9.9pp** 0.384 0.001 
Race/Ethnicity (I16sch) (29 sch) 


*Statistically significant at p < .05; **Statistically significant at p < .01. 
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Table D.2. Subgroup Analysis for Taking College Credit Plus Courses 


Unadj. Taal ex=\oas Effect Size 
Population (ej ¢-Ce (= M-0"/-) | cs 1 a) ola nlo | Comp Mean Estimate | (Cox’s Index) 


20,929 41,628 
All Students Grades 10-12 20.7% 13.4% +7.3pp*** 0.318 0.000 
(16 sch) (32 sch) 


Economically 12,758 24,685 


Grades 10-12 16.2% 71% +9.1pp*** 0.560 0.000 
Disadvantaged (16 sch) (32 sch) 7 Pe 

Not Economically 8,171 15,886 

: Grades 10-12 27.6% 22.8% +4.8pp 0.155 0.123 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 9,928 20,126 

=, Grades 10-12 14.8% 7.2% +7.7pp*** 0.494 0.000 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 11,001 21,458 

s Grades 10-12 27.3% 19.2% +8.1pp** 0.277 0.001 
Race/Ethnicity (l6sch) (29 sch) 


13,216 26,638 
All Students Grades 11-12 25.5% 18.5% +7.0pp** 0.248 0.004 
(16 sch) (32 sch) 


Economically 7,777 15,515 
. Grades 11-12 19.1% 9.6% +9.5pp*** 0.483 0.000 
Disadvantaged (16 sch) (32 sch) 
Not Economically 5,439 10,453 
. Grades 11-12 36.3% 31.2% +5.0pp 0.137 0.171 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 6,162 12,879 
_ Grades 11-12 18.0% 9.7% +8.3pp** 0.431 0.001 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 7,054 13,732 
Ww Grades 11-12 35.3% 26.7% +8.6pp** 0.245 0.004 
Race/Ethnicity (I16sch) (29 sch) 


7,713 14,990 
All Students Grade 10 12.0% 4.2% +7.8pp*** 0.684 0.000 
(16 sch) (32 sch) 


Economically 4,981 9,170 
. Grade 10 10.5% 2.9% +7.6pp*** 0.832 0.000 
Disadvantaged (16 sch) (32 sch) 
Not Economically 2,732 5,433 
; Grade 10 12.4% 6.6% +5.9pp 0.427 0.127 
Disadvantaged (12 sch) (19 sch) 
Underrepresented Race/ 3,766 7,247 
. Grade 10 9.7% 2.6% +7.1pp*** 0.851 0.000 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 3,947 7,726 
oF Grade 10 14.4% 5.8% +8.5pp** 0.605 0.003 
Race/Ethnicity (I16sch) (29 sch) 


Statistically significant at p < .01; ***Statistically significant at p < .001. 
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Table D.3. Subgroup Analysis for Taking College Credit Plus Courses 


Unadj. Taal ex=\oas Effect Size 
Population (ej ¢-Ce (= M-0"/-) (i cs 1) ola nl ody | (ovoy nn) oy (=s-18) Estimate | (Cox’s Index) 


20,929 41,628 
All Students Grades 10-12 14.5% 13.5% +1.0pp 0.052 0.653 
(16 sch) (32 sch) 


Economically 12,758 24,685 
: Grades 10-12 10.6% 10.1% +0.5pp 0.032 0.826 
Disadvantaged (16 sch) (32 sch) 
Not Economically 8,171 15,886 
. Grades 10-12 24.7% 17.7% +7.1pp* 0.258 0.019 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 9,928 20,126 
Es Grades 10-12 10.0% 9.6% +0.4pp 0.025 0.856 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 11,001 21,458 
_ Grades 10-12 19.4% 171% +2.2pp 0.091 0.445 
Race/Ethnicity (l6sch) (29 sch) 


13,216 26,638 
All Students Grades 11-12 17.9% 16.6% +1.3pp 0.053 0.662 
(16 sch) (32 sch) 


Economically 7,777 15,515 
. Grades 11-12 13.5% 12.3% +1.2pp 0.064 0.672 
Disadvantaged (16 sch) (32 sch) 
Not Economically 5,439 10,453 
. Grades 11-12 28.8% 21.8% +7.1pp 0.227 0.065 
Disadvantaged (12 sch) (21 sch) 
Underrepresented Race/ 6,162 12,879 
. Grades 11-12 13.3% 11.9% +1.4pp 0.075 0.596 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 7,054. 13,732 
Grades 11-12 23.0% 21.1% +1.9pp 0.066 0.617 
Race/Ethnicity (Il6sch)  (29sch) 
7,713 14,990 
All Students Grade 10 (16 sch) (32 sch) 8.5% 7.9% +0.6pp 0.051 0.730 
Economically 4,981 9,170 
A Grade 10 6.0% 6.4% -0.4pp -0.042 0.819 
Disadvantaged (16 sch) (32 sch) 
Not Economically 2,732 5,433 
. Grade 10 17.0% 9.8% +7.2pp* 0.382 0.016 
Disadvantaged (12 sch) (19 sch) 
Underrepresented Race/ 3,766 7,247 
an Grade 10 5.2% 5.5% -0.3pp -0.042 0.820 
Ethnicity (16 sch) (32 sch) 
Not Underrepresented 3,947 7,726 
a Grade 10 13.0% 10.1% +2.9pp 0.175 0.226 
Race/Ethnicity (l6sch)  (29sch) 


*Statistically significant at p < .05 
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